2020-07-13T05:50:22.340 INFO:root:teuthology version: 1.0.0-da68153 2020-07-13T05:50:22.343 DEBUG:teuthology.run:Teuthology command: teuthology --block --name teuthology-2020-07-13_03:05:02-rgw-master-distro-basic-smithi --archive /home/teuthworker/archive/teuthology-2020-07-13_03:05:02-rgw-master-distro-basic-smithi/5218593 --description rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} --verbose --lock --owner scheduled_teuthology@teuthology -- /tmp/teuthology-worker.hbiqdx8s.tmp 2020-07-13T05:50:22.373 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-07-13T05:50:22.432 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2020-07-13_03:05:02-rgw-master-distro-basic-smithi/5218593 branch: master description: rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: ceph-qa@ceph.io first_in_suite: false job_id: '5218593' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2020-07-13_03:05:02-rgw-master-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: master ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync obj etag verify: true setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) sha1: 1ac5c74502da6dc8ad965625df452eb42a846489 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 1ac5c74502da6dc8ad965625df452eb42a846489 rgw: compression type: random frontend: civetweb rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: master sha1: 1ac5c74502da6dc8ad965625df452eb42a846489 owner: scheduled_teuthology@teuthology priority: 1000 repo: git://git.ceph.com/ceph.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 1ac5c74502da6dc8ad965625df452eb42a846489 sleep_before_teardown: 0 suite: rgw suite_branch: master suite_path: /home/teuthworker/src/git.ceph.com_ceph_master/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph.git suite_sha1: 1ac5c74502da6dc8ad965625df452eb42a846489 tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master tube: smithi verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.20773 2020-07-13T05:50:22.433 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_master/qa; will attempt to use it 2020-07-13T05:50:22.434 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks 2020-07-13T05:50:22.434 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-07-13T05:50:22.436 INFO:teuthology.task.internal:Checking packages... 2020-07-13T05:50:22.459 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '1ac5c74502da6dc8ad965625df452eb42a846489' 2020-07-13T05:50:22.459 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-07-13T05:50:22.460 INFO:teuthology.packaging:ref: None 2020-07-13T05:50:22.460 INFO:teuthology.packaging:tag: None 2020-07-13T05:50:22.460 INFO:teuthology.packaging:branch: master 2020-07-13T05:50:22.461 INFO:teuthology.packaging:sha1: 1ac5c74502da6dc8ad965625df452eb42a846489 2020-07-13T05:50:22.461 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=master 2020-07-13T05:50:23.141 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-3402-g1ac5c74-1bionic 2020-07-13T05:50:23.142 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-07-13T05:50:23.152 INFO:teuthology.task.internal:no buildpackages task found 2020-07-13T05:50:23.152 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-07-13T05:50:23.169 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-07-13T05:50:23.170 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-07-13T05:50:23.233 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:50:33.255 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:50:43.271 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:50:53.288 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:51:03.308 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:51:13.323 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:51:23.340 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:51:33.360 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:51:43.378 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:51:53.389 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:52:03.406 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:52:13.423 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:52:23.439 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:52:33.462 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:52:43.480 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:52:53.495 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:53:03.513 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:53:13.532 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:53:23.551 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:53:33.577 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:53:43.596 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:53:53.610 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:54:03.629 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:54:13.647 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:54:23.671 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:54:33.685 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:54:43.706 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:54:53.724 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:55:03.744 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:55:13.757 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:55:23.780 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:55:33.801 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:55:43.820 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:55:53.838 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:56:03.856 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:56:13.875 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:56:23.896 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:56:33.917 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:56:43.938 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:56:53.956 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:57:03.985 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:57:14.003 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:57:24.021 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:57:34.042 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T05:57:44.060 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:57:54.081 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:58:04.102 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:58:14.118 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:58:24.142 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:58:34.164 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:58:44.183 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:58:54.205 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:59:04.225 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:59:14.241 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:59:24.274 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:59:34.293 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:59:44.321 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T05:59:54.356 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T06:00:04.380 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T06:00:14.399 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T06:00:24.420 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T06:00:34.440 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-13T06:00:44.463 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T06:00:54.486 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T06:01:04.503 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-13T06:01:14.519 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_teuthology@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/teuthology-2020-07-13_03:05:02-rgw-master-distro-basic-smithi/5218593'} 2020-07-13T06:01:14.553 DEBUG:teuthology.lock.ops:locked ubuntu@smithi017.front.sepia.ceph.com, ubuntu@smithi014.front.sepia.ceph.com 2020-07-13T06:01:14.619 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6d4841bd08>, signals=[15]) 2020-07-13T06:01:14.815 INFO:teuthology.provision.fog.smithi017:Scheduling deploy of ubuntu 18.04 2020-07-13T06:01:14.824 INFO:teuthology.provision.fog.smithi014:Scheduling deploy of ubuntu 18.04 2020-07-13T06:01:15.147 INFO:teuthology.orchestra.console:Power off smithi014 2020-07-13T06:01:15.148 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi014.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-07-13T06:01:15.174 INFO:teuthology.orchestra.console:Power off smithi017 2020-07-13T06:01:15.175 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-07-13T06:01:15.186 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi014.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-13T06:01:15.202 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-13T06:01:19.218 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi014.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-13T06:01:19.248 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-13T06:01:23.336 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi014.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-13T06:01:23.363 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-13T06:01:27.459 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi014.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-13T06:01:27.483 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-13T06:01:27.679 INFO:teuthology.orchestra.console:Power off for smithi014 completed 2020-07-13T06:01:27.698 INFO:teuthology.orchestra.console:Power off for smithi017 completed 2020-07-13T06:01:27.781 INFO:teuthology.orchestra.console:Power on smithi014 2020-07-13T06:01:27.782 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi014.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-07-13T06:01:27.799 INFO:teuthology.orchestra.console:Power on smithi017 2020-07-13T06:01:27.800 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-07-13T06:01:27.814 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi014.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-13T06:01:27.829 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-13T06:01:31.848 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi014.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-13T06:01:31.867 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-13T06:01:32.070 INFO:teuthology.orchestra.console:Power on for smithi014 completed 2020-07-13T06:01:32.171 INFO:teuthology.provision.fog.smithi014:Waiting for deploy to finish 2020-07-13T06:01:35.983 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-13T06:01:36.205 INFO:teuthology.orchestra.console:Power on for smithi017 completed 2020-07-13T06:01:36.306 INFO:teuthology.provision.fog.smithi017:Waiting for deploy to finish 2020-07-13T06:04:18.345 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi014.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:04:22.401 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:04:27.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi014.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:04:31.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:04:36.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi014.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:04:40.531 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:04:45.519 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi014.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:04:49.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:04:54.579 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi014.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:04:58.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:05:03.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi014.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:05:07.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:06:09.690 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi014.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:06:09.766 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-07-13T06:06:09.769 INFO:teuthology.orchestra.run.smithi014:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-07-13T06:06:10.115 INFO:teuthology.provision.fog.smithi014:Node is ready 2020-07-13T06:06:10.116 INFO:teuthology.orchestra.run.smithi014:> hostname 2020-07-13T06:06:10.162 INFO:teuthology.orchestra.run.smithi014.stdout:smithi014.front.sepia.ceph.com 2020-07-13T06:06:10.163 INFO:teuthology.orchestra.run.smithi014:> grep smithi014.front.sepia.ceph.com /etc/hosts 2020-07-13T06:06:10.210 INFO:teuthology.orchestra.run.smithi014.stdout:172.21.15.14 smithi014.front.sepia.ceph.com smithi014 2020-07-13T06:06:10.211 INFO:teuthology.orchestra.run.smithi014:> sudo hostname smithi014 2020-07-13T06:06:10.265 INFO:teuthology.orchestra.run.smithi014:> sudo sed -i -e 's/smithi014.front.sepia.ceph.com/smithi014/g' /etc/hosts 2020-07-13T06:06:10.324 INFO:teuthology.orchestra.run.smithi014:> sudo sed -i -e 's/172.21.15.14/172.21.15.14/g' /etc/hosts 2020-07-13T06:06:10.380 INFO:teuthology.orchestra.run.smithi014:> sudo sed -i -e 's/smithi014.front.sepia.ceph.com/smithi014/g' /etc/hostname 2020-07-13T06:06:10.436 INFO:teuthology.orchestra.run.smithi014:> sudo hostname smithi014 2020-07-13T06:06:10.493 INFO:teuthology.provision.fog.smithi014:Deploy complete! 2020-07-13T06:06:10.494 DEBUG:teuthology.parallel:result is None 2020-07-13T06:06:13.703 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:06:13.783 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-07-13T06:06:13.783 INFO:teuthology.orchestra.run.smithi017:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-07-13T06:06:14.115 INFO:teuthology.provision.fog.smithi017:Node is ready 2020-07-13T06:06:14.116 INFO:teuthology.orchestra.run.smithi017:> hostname 2020-07-13T06:06:14.167 INFO:teuthology.orchestra.run.smithi017.stdout:smithi017.front.sepia.ceph.com 2020-07-13T06:06:14.168 INFO:teuthology.orchestra.run.smithi017:> grep smithi017.front.sepia.ceph.com /etc/hosts 2020-07-13T06:06:14.217 INFO:teuthology.orchestra.run.smithi017.stdout:172.21.15.17 smithi017.front.sepia.ceph.com smithi017 2020-07-13T06:06:14.218 INFO:teuthology.orchestra.run.smithi017:> sudo hostname smithi017 2020-07-13T06:06:14.276 INFO:teuthology.orchestra.run.smithi017:> sudo sed -i -e 's/smithi017.front.sepia.ceph.com/smithi017/g' /etc/hosts 2020-07-13T06:06:14.333 INFO:teuthology.orchestra.run.smithi017:> sudo sed -i -e 's/172.21.15.17/172.21.15.17/g' /etc/hosts 2020-07-13T06:06:14.390 INFO:teuthology.orchestra.run.smithi017:> sudo sed -i -e 's/smithi017.front.sepia.ceph.com/smithi017/g' /etc/hostname 2020-07-13T06:06:14.445 INFO:teuthology.orchestra.run.smithi017:> sudo hostname smithi017 2020-07-13T06:06:14.500 INFO:teuthology.provision.fog.smithi017:Deploy complete! 2020-07-13T06:06:14.501 DEBUG:teuthology.parallel:result is None 2020-07-13T06:06:14.645 INFO:teuthology.lock.ops:Checking smithi017.front.sepia.ceph.com 2020-07-13T06:06:14.646 INFO:teuthology.lock.ops:New key found. Updating... 2020-07-13T06:06:14.671 INFO:teuthology.lock.ops:Checking smithi014.front.sepia.ceph.com 2020-07-13T06:06:14.672 INFO:teuthology.lock.ops:New key found. Updating... 2020-07-13T06:06:14.693 INFO:teuthology.lock.ops:Updating [smithi017.front.sepia.ceph.com]: set os type and version on server 2020-07-13T06:06:14.694 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-07-13T06:06:14.695 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:06:14.775 INFO:teuthology.orchestra.run.smithi017:> true 2020-07-13T06:06:15.026 INFO:teuthology.orchestra.run.smithi017:> uname -m 2020-07-13T06:06:15.072 INFO:teuthology.orchestra.run.smithi017.stdout:x86_64 2020-07-13T06:06:15.073 INFO:teuthology.orchestra.run.smithi017:> cat /etc/os-release 2020-07-13T06:06:15.119 INFO:teuthology.orchestra.run.smithi017.stdout:NAME="Ubuntu" 2020-07-13T06:06:15.120 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-07-13T06:06:15.120 INFO:teuthology.orchestra.run.smithi017.stdout:ID=ubuntu 2020-07-13T06:06:15.120 INFO:teuthology.orchestra.run.smithi017.stdout:ID_LIKE=debian 2020-07-13T06:06:15.120 INFO:teuthology.orchestra.run.smithi017.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-07-13T06:06:15.120 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_ID="18.04" 2020-07-13T06:06:15.121 INFO:teuthology.orchestra.run.smithi017.stdout:HOME_URL="https://www.ubuntu.com/" 2020-07-13T06:06:15.121 INFO:teuthology.orchestra.run.smithi017.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-07-13T06:06:15.121 INFO:teuthology.orchestra.run.smithi017.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-07-13T06:06:15.121 INFO:teuthology.orchestra.run.smithi017.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-07-13T06:06:15.122 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_CODENAME=bionic 2020-07-13T06:06:15.122 INFO:teuthology.orchestra.run.smithi017.stdout:UBUNTU_CODENAME=bionic 2020-07-13T06:06:15.123 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server 2020-07-13T06:06:15.146 INFO:teuthology.lock.ops:Updating [smithi014.front.sepia.ceph.com]: set os type and version on server 2020-07-13T06:06:15.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-07-13T06:06:15.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi014.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:06:15.224 INFO:teuthology.orchestra.run.smithi014:> true 2020-07-13T06:06:15.466 INFO:teuthology.orchestra.run.smithi014:> uname -m 2020-07-13T06:06:15.514 INFO:teuthology.orchestra.run.smithi014.stdout:x86_64 2020-07-13T06:06:15.514 INFO:teuthology.orchestra.run.smithi014:> cat /etc/os-release 2020-07-13T06:06:15.562 INFO:teuthology.orchestra.run.smithi014.stdout:NAME="Ubuntu" 2020-07-13T06:06:15.562 INFO:teuthology.orchestra.run.smithi014.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-07-13T06:06:15.563 INFO:teuthology.orchestra.run.smithi014.stdout:ID=ubuntu 2020-07-13T06:06:15.563 INFO:teuthology.orchestra.run.smithi014.stdout:ID_LIKE=debian 2020-07-13T06:06:15.563 INFO:teuthology.orchestra.run.smithi014.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-07-13T06:06:15.563 INFO:teuthology.orchestra.run.smithi014.stdout:VERSION_ID="18.04" 2020-07-13T06:06:15.563 INFO:teuthology.orchestra.run.smithi014.stdout:HOME_URL="https://www.ubuntu.com/" 2020-07-13T06:06:15.564 INFO:teuthology.orchestra.run.smithi014.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-07-13T06:06:15.564 INFO:teuthology.orchestra.run.smithi014.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-07-13T06:06:15.564 INFO:teuthology.orchestra.run.smithi014.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-07-13T06:06:15.564 INFO:teuthology.orchestra.run.smithi014.stdout:VERSION_CODENAME=bionic 2020-07-13T06:06:15.564 INFO:teuthology.orchestra.run.smithi014.stdout:UBUNTU_CODENAME=bionic 2020-07-13T06:06:15.565 INFO:teuthology.lock.ops:Updating smithi014.front.sepia.ceph.com on lock server 2020-07-13T06:06:15.585 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-07-13T06:06:15.620 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi014.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmQcdwKZF703ribUpzgoiJtFPNwO6YJpt3QdUhjrgrYsyMrNdTc6ePfhsPyQiUS7ch6f83BQanIliWLoIfxdHNHrOSdPtLtj569MND1/HJAnCfRm7Abc26MHr63Ej2ZrA3yfLTNz4p6na1Gm4wo0N3jBFuIP9kuXZeHoeTjIuU01e8Hcqmacb3ptSYlAaoqkblDgblem+kz6eF1yXiNCEydEgT2GKZ9djV/w8zYRrsWBu1b6J+Rq2Tn/Jp1XYFupOMo4Bpi6Ztm8GSlc0sMo0nETDec3Mwgpf/wcG65bYFVHkPPIvPl2u6VZ5yE0qaO+zPFz409WyM4gRPRkZV5lPL smithi017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5Tg3xFvlmPJANVeH97ibsyYtT0l3duWosTCQdsiAKn2wAntdiIolQt5T5sNUSlPRj+sUH7eK5AF/+lijwzESp3GGzDRFrFoaWRbWOZeCGjTUUIKFFhlrf3hcXZCVQj9+Fz/1mUiZiM6dRnNEvvrFt7485R3OGrVpU6/QWvggV8Mqv/bLIerFbS1Z+3pD8jIFuepUKMV5IaHyXQrzmWESGmyR+cJYGu2CmjNEovrX3jzF9xCx+wFysKyorm0WzbveD4dQ+OyLixMa7xyO8/1i7nolBBmFltVOcYNuKeXTdRRZr+Hna3U/RutVzRuSPcOaXB6q39zmAeMC+6OIOuBz9 2020-07-13T06:06:15.622 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-07-13T06:06:15.678 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-07-13T06:06:15.692 INFO:teuthology.task.internal:Saving configuration 2020-07-13T06:06:15.706 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-07-13T06:06:15.716 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-07-13T06:06:15.733 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-07-13 06:01:14.528461', 'description': '/home/teuthworker/archive/teuthology-2020-07-13_03:05:02-rgw-master-distro-basic-smithi/5218593', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOzRq4jj7pKSy14lG+RVhuBkh2nXLLgsVwxkZV8qu2fs', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi017.front.sepia.ceph.com'} 2020-07-13T06:06:15.751 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-07-13 06:01:14.528481', 'description': '/home/teuthworker/archive/teuthology-2020-07-13_03:05:02-rgw-master-distro-basic-smithi/5218593', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJYWoNVfN+gYSTsQozlxBSvsPJOTz5Bs3ndwazFtaOky', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi014.front.sepia.ceph.com'} 2020-07-13T06:06:15.752 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-07-13T06:06:15.763 INFO:teuthology.task.internal:roles: ubuntu@smithi017.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2020-07-13T06:06:15.763 INFO:teuthology.task.internal:roles: ubuntu@smithi014.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2020-07-13T06:06:15.764 INFO:teuthology.run_tasks:Running task console_log... 2020-07-13T06:06:15.849 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6d4c31e2f0>, signals=[15]) 2020-07-13T06:06:15.850 INFO:teuthology.run_tasks:Running task internal.connect... 2020-07-13T06:06:15.861 INFO:teuthology.task.internal:Opening connections... 2020-07-13T06:06:15.861 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi017.front.sepia.ceph.com 2020-07-13T06:06:15.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:06:15.967 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi014.front.sepia.ceph.com 2020-07-13T06:06:15.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi014.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:06:16.046 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-07-13T06:06:16.057 INFO:teuthology.orchestra.run.smithi017:> uname -m 2020-07-13T06:06:16.176 INFO:teuthology.orchestra.run.smithi017.stdout:x86_64 2020-07-13T06:06:16.177 INFO:teuthology.orchestra.run.smithi017:> cat /etc/os-release 2020-07-13T06:06:16.223 INFO:teuthology.orchestra.run.smithi017.stdout:NAME="Ubuntu" 2020-07-13T06:06:16.224 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-07-13T06:06:16.224 INFO:teuthology.orchestra.run.smithi017.stdout:ID=ubuntu 2020-07-13T06:06:16.224 INFO:teuthology.orchestra.run.smithi017.stdout:ID_LIKE=debian 2020-07-13T06:06:16.225 INFO:teuthology.orchestra.run.smithi017.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-07-13T06:06:16.225 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_ID="18.04" 2020-07-13T06:06:16.225 INFO:teuthology.orchestra.run.smithi017.stdout:HOME_URL="https://www.ubuntu.com/" 2020-07-13T06:06:16.226 INFO:teuthology.orchestra.run.smithi017.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-07-13T06:06:16.226 INFO:teuthology.orchestra.run.smithi017.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-07-13T06:06:16.226 INFO:teuthology.orchestra.run.smithi017.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-07-13T06:06:16.227 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_CODENAME=bionic 2020-07-13T06:06:16.228 INFO:teuthology.orchestra.run.smithi017.stdout:UBUNTU_CODENAME=bionic 2020-07-13T06:06:16.228 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server 2020-07-13T06:06:16.248 INFO:teuthology.orchestra.run.smithi014:> uname -m 2020-07-13T06:06:16.254 INFO:teuthology.orchestra.run.smithi014.stdout:x86_64 2020-07-13T06:06:16.255 INFO:teuthology.orchestra.run.smithi014:> cat /etc/os-release 2020-07-13T06:06:16.312 INFO:teuthology.orchestra.run.smithi014.stdout:NAME="Ubuntu" 2020-07-13T06:06:16.313 INFO:teuthology.orchestra.run.smithi014.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-07-13T06:06:16.313 INFO:teuthology.orchestra.run.smithi014.stdout:ID=ubuntu 2020-07-13T06:06:16.313 INFO:teuthology.orchestra.run.smithi014.stdout:ID_LIKE=debian 2020-07-13T06:06:16.314 INFO:teuthology.orchestra.run.smithi014.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-07-13T06:06:16.314 INFO:teuthology.orchestra.run.smithi014.stdout:VERSION_ID="18.04" 2020-07-13T06:06:16.314 INFO:teuthology.orchestra.run.smithi014.stdout:HOME_URL="https://www.ubuntu.com/" 2020-07-13T06:06:16.314 INFO:teuthology.orchestra.run.smithi014.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-07-13T06:06:16.314 INFO:teuthology.orchestra.run.smithi014.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-07-13T06:06:16.315 INFO:teuthology.orchestra.run.smithi014.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-07-13T06:06:16.315 INFO:teuthology.orchestra.run.smithi014.stdout:VERSION_CODENAME=bionic 2020-07-13T06:06:16.315 INFO:teuthology.orchestra.run.smithi014.stdout:UBUNTU_CODENAME=bionic 2020-07-13T06:06:16.315 INFO:teuthology.lock.ops:Updating smithi014.front.sepia.ceph.com on lock server 2020-07-13T06:06:16.338 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-07-13T06:06:16.889 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-07-13T06:06:16.908 INFO:teuthology.task.internal:Checking for old test directory... 2020-07-13T06:06:16.909 INFO:teuthology.orchestra.run.smithi014:> test '!' -e /home/ubuntu/cephtest 2020-07-13T06:06:16.919 INFO:teuthology.orchestra.run.smithi017:> test '!' -e /home/ubuntu/cephtest 2020-07-13T06:06:16.925 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-07-13T06:06:16.933 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-07-13T06:06:16.933 INFO:teuthology.orchestra.run.smithi014:> test -z $(ls -A /var/lib/ceph) 2020-07-13T06:06:16.966 INFO:teuthology.orchestra.run.smithi017:> test -z $(ls -A /var/lib/ceph) 2020-07-13T06:06:16.974 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-07-13T06:06:17.023 INFO:teuthology.run_tasks:Running task kernel... 2020-07-13T06:06:17.042 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2020-07-13T06:06:17.043 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-07-13T06:06:17.044 INFO:teuthology.orchestra.run.smithi017:> uname -r 2020-07-13T06:06:17.050 INFO:teuthology.orchestra.run.smithi017.stdout:4.15.0-109-generic 2020-07-13T06:06:17.050 INFO:teuthology.task.kernel:Running kernel on smithi017: 4.15.0-109-generic 2020-07-13T06:06:17.051 INFO:teuthology.orchestra.run.smithi017:> sudo apt-get clean 2020-07-13T06:06:17.147 INFO:teuthology.orchestra.run.smithi017:> sudo apt-get update 2020-07-13T06:06:17.317 INFO:teuthology.orchestra.run.smithi017.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-07-13T06:06:17.439 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-07-13T06:06:17.518 INFO:teuthology.orchestra.run.smithi017.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-07-13T06:06:17.784 INFO:teuthology.orchestra.run.smithi017.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-07-13T06:06:18.136 INFO:teuthology.orchestra.run.smithi017.stdout:Fetched 252 kB in 1s (337 kB/s) 2020-07-13T06:06:19.755 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2020-07-13T06:06:19.801 INFO:teuthology.orchestra.run.smithi017:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-07-13T06:06:19.883 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2020-07-13T06:06:20.140 INFO:teuthology.orchestra.run.smithi017.stdout:Building dependency tree... 2020-07-13T06:06:20.141 INFO:teuthology.orchestra.run.smithi017.stdout:Reading state information... 2020-07-13T06:06:20.231 INFO:teuthology.orchestra.run.smithi017.stderr:E: Unable to locate package linux-image-current-generic 2020-07-13T06:06:20.233 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-07-13T06:06:20.234 INFO:teuthology.orchestra.run.smithi017:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-07-13T06:06:20.312 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2020-07-13T06:06:20.544 INFO:teuthology.orchestra.run.smithi017.stdout:Building dependency tree... 2020-07-13T06:06:20.545 INFO:teuthology.orchestra.run.smithi017.stdout:Reading state information... 2020-07-13T06:06:20.761 INFO:teuthology.orchestra.run.smithi017.stdout:linux-image-generic is already the newest version (4.15.0.109.97). 2020-07-13T06:06:20.762 INFO:teuthology.orchestra.run.smithi017.stdout:linux-image-generic set to manually installed. 2020-07-13T06:06:20.762 INFO:teuthology.orchestra.run.smithi017.stdout:0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 2020-07-13T06:06:20.763 INFO:teuthology.orchestra.run.smithi017:> dpkg -s linux-image-generic 2020-07-13T06:06:20.786 INFO:teuthology.orchestra.run.smithi017.stdout:Package: linux-image-generic 2020-07-13T06:06:20.787 INFO:teuthology.orchestra.run.smithi017.stdout:Status: install ok installed 2020-07-13T06:06:20.787 INFO:teuthology.orchestra.run.smithi017.stdout:Priority: optional 2020-07-13T06:06:20.787 INFO:teuthology.orchestra.run.smithi017.stdout:Section: kernel 2020-07-13T06:06:20.788 INFO:teuthology.orchestra.run.smithi017.stdout:Installed-Size: 16 2020-07-13T06:06:20.788 INFO:teuthology.orchestra.run.smithi017.stdout:Maintainer: Ubuntu Kernel Team 2020-07-13T06:06:20.788 INFO:teuthology.orchestra.run.smithi017.stdout:Architecture: amd64 2020-07-13T06:06:20.789 INFO:teuthology.orchestra.run.smithi017.stdout:Source: linux-meta 2020-07-13T06:06:20.789 INFO:teuthology.orchestra.run.smithi017.stdout:Version: 4.15.0.109.97 2020-07-13T06:06:20.789 INFO:teuthology.orchestra.run.smithi017.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.9) 2020-07-13T06:06:20.790 INFO:teuthology.orchestra.run.smithi017.stdout:Depends: linux-image-4.15.0-109-generic, linux-modules-extra-4.15.0-109-generic, linux-firmware, intel-microcode, amd64-microcode 2020-07-13T06:06:20.790 INFO:teuthology.orchestra.run.smithi017.stdout:Recommends: thermald 2020-07-13T06:06:20.790 INFO:teuthology.orchestra.run.smithi017.stdout:Description: Generic Linux kernel image 2020-07-13T06:06:20.791 INFO:teuthology.orchestra.run.smithi017.stdout: This package will always depend on the latest generic kernel image 2020-07-13T06:06:20.791 INFO:teuthology.orchestra.run.smithi017.stdout: available. 2020-07-13T06:06:20.792 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-07-13T06:06:20.792 INFO:teuthology.orchestra.run.smithi014:> uname -r 2020-07-13T06:06:20.799 INFO:teuthology.orchestra.run.smithi014.stdout:4.15.0-109-generic 2020-07-13T06:06:20.799 INFO:teuthology.task.kernel:Running kernel on smithi014: 4.15.0-109-generic 2020-07-13T06:06:20.800 INFO:teuthology.orchestra.run.smithi014:> sudo apt-get clean 2020-07-13T06:06:20.894 INFO:teuthology.orchestra.run.smithi014:> sudo apt-get update 2020-07-13T06:06:21.156 INFO:teuthology.orchestra.run.smithi014.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-07-13T06:06:21.157 INFO:teuthology.orchestra.run.smithi014.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-07-13T06:06:21.219 INFO:teuthology.orchestra.run.smithi014.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-07-13T06:06:21.485 INFO:teuthology.orchestra.run.smithi014.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-07-13T06:06:21.888 INFO:teuthology.orchestra.run.smithi014.stdout:Fetched 252 kB in 1s (334 kB/s) 2020-07-13T06:06:23.448 INFO:teuthology.orchestra.run.smithi014.stdout:Reading package lists... 2020-07-13T06:06:23.497 INFO:teuthology.orchestra.run.smithi014:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-07-13T06:06:23.584 INFO:teuthology.orchestra.run.smithi014.stdout:Reading package lists... 2020-07-13T06:06:23.836 INFO:teuthology.orchestra.run.smithi014.stdout:Building dependency tree... 2020-07-13T06:06:23.837 INFO:teuthology.orchestra.run.smithi014.stdout:Reading state information... 2020-07-13T06:06:23.926 INFO:teuthology.orchestra.run.smithi014.stderr:E: Unable to locate package linux-image-current-generic 2020-07-13T06:06:23.927 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-07-13T06:06:23.929 INFO:teuthology.orchestra.run.smithi014:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-07-13T06:06:24.017 INFO:teuthology.orchestra.run.smithi014.stdout:Reading package lists... 2020-07-13T06:06:24.248 INFO:teuthology.orchestra.run.smithi014.stdout:Building dependency tree... 2020-07-13T06:06:24.248 INFO:teuthology.orchestra.run.smithi014.stdout:Reading state information... 2020-07-13T06:06:24.466 INFO:teuthology.orchestra.run.smithi014.stdout:linux-image-generic is already the newest version (4.15.0.109.97). 2020-07-13T06:06:24.467 INFO:teuthology.orchestra.run.smithi014.stdout:linux-image-generic set to manually installed. 2020-07-13T06:06:24.467 INFO:teuthology.orchestra.run.smithi014.stdout:0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 2020-07-13T06:06:24.468 INFO:teuthology.orchestra.run.smithi014:> dpkg -s linux-image-generic 2020-07-13T06:06:24.492 INFO:teuthology.orchestra.run.smithi014.stdout:Package: linux-image-generic 2020-07-13T06:06:24.492 INFO:teuthology.orchestra.run.smithi014.stdout:Status: install ok installed 2020-07-13T06:06:24.493 INFO:teuthology.orchestra.run.smithi014.stdout:Priority: optional 2020-07-13T06:06:24.493 INFO:teuthology.orchestra.run.smithi014.stdout:Section: kernel 2020-07-13T06:06:24.493 INFO:teuthology.orchestra.run.smithi014.stdout:Installed-Size: 16 2020-07-13T06:06:24.494 INFO:teuthology.orchestra.run.smithi014.stdout:Maintainer: Ubuntu Kernel Team 2020-07-13T06:06:24.494 INFO:teuthology.orchestra.run.smithi014.stdout:Architecture: amd64 2020-07-13T06:06:24.495 INFO:teuthology.orchestra.run.smithi014.stdout:Source: linux-meta 2020-07-13T06:06:24.495 INFO:teuthology.orchestra.run.smithi014.stdout:Version: 4.15.0.109.97 2020-07-13T06:06:24.496 INFO:teuthology.orchestra.run.smithi014.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.9) 2020-07-13T06:06:24.496 INFO:teuthology.orchestra.run.smithi014.stdout:Depends: linux-image-4.15.0-109-generic, linux-modules-extra-4.15.0-109-generic, linux-firmware, intel-microcode, amd64-microcode 2020-07-13T06:06:24.496 INFO:teuthology.orchestra.run.smithi014.stdout:Recommends: thermald 2020-07-13T06:06:24.496 INFO:teuthology.orchestra.run.smithi014.stdout:Description: Generic Linux kernel image 2020-07-13T06:06:24.497 INFO:teuthology.orchestra.run.smithi014.stdout: This package will always depend on the latest generic kernel image 2020-07-13T06:06:24.497 INFO:teuthology.orchestra.run.smithi014.stdout: available. 2020-07-13T06:06:24.497 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-07-13T06:06:24.498 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2020-07-13T06:06:24.498 INFO:teuthology.orchestra.run.smithi017:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-07-13T06:06:24.531 INFO:teuthology.orchestra.run.smithi017.stdout:ttyS1 2020-07-13T06:06:24.551 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2020-07-13T06:06:24.552 INFO:teuthology.orchestra.run.smithi014:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-07-13T06:06:24.569 INFO:teuthology.orchestra.run.smithi014.stdout:ttyS1 2020-07-13T06:06:24.590 INFO:teuthology.run_tasks:Running task internal.base... 2020-07-13T06:06:24.601 INFO:teuthology.task.internal:Creating test directory... 2020-07-13T06:06:24.602 INFO:teuthology.orchestra.run.smithi014:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-07-13T06:06:24.634 INFO:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-07-13T06:06:24.641 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-07-13T06:06:24.652 INFO:teuthology.run_tasks:Running task internal.archive... 2020-07-13T06:06:24.666 INFO:teuthology.task.internal:Creating archive directory... 2020-07-13T06:06:24.666 INFO:teuthology.orchestra.run.smithi014:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-07-13T06:06:24.682 INFO:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-07-13T06:06:24.691 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-07-13T06:06:24.701 INFO:teuthology.task.internal:Enabling coredump saving... 2020-07-13T06:06:24.702 INFO:teuthology.orchestra.run.smithi014:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-07-13T06:06:24.734 INFO:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-07-13T06:06:24.756 INFO:teuthology.orchestra.run.smithi014.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-07-13T06:06:24.762 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-07-13T06:06:24.763 INFO:teuthology.orchestra.run.smithi014.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-07-13T06:06:24.769 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-07-13T06:06:24.770 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-07-13T06:06:24.779 INFO:teuthology.task.internal:Configuring sudo... 2020-07-13T06:06:24.780 INFO:teuthology.orchestra.run.smithi014:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-07-13T06:06:24.814 INFO:teuthology.orchestra.run.smithi017:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-07-13T06:06:24.829 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-07-13T06:06:24.841 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-07-13T06:06:24.842 INFO:teuthology.orchestra.run.smithi014:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-07-13T06:06:24.874 INFO:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-07-13T06:06:24.881 INFO:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-07-13T06:06:24.928 INFO:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-07-13T06:06:24.976 INFO:teuthology.orchestra.run.smithi017:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-07-13T06:06:25.064 INFO:teuthology.orchestra.run.smithi014:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-07-13T06:06:25.071 INFO:teuthology.orchestra.run.smithi014:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-07-13T06:06:25.118 INFO:teuthology.orchestra.run.smithi014:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-07-13T06:06:25.206 INFO:teuthology.orchestra.run.smithi014:> sudo service rsyslog restart 2020-07-13T06:06:25.250 INFO:teuthology.orchestra.run.smithi017:> sudo service rsyslog restart 2020-07-13T06:06:25.293 INFO:teuthology.run_tasks:Running task internal.timer... 2020-07-13T06:06:25.305 INFO:teuthology.task.internal:Starting timer... 2020-07-13T06:06:25.305 INFO:teuthology.run_tasks:Running task pcp... 2020-07-13T06:06:25.320 INFO:teuthology.run_tasks:Running task selinux... 2020-07-13T06:06:25.332 DEBUG:teuthology.task.selinux:Excluding smithi017: OS 'ubuntu' does not support SELinux 2020-07-13T06:06:25.333 DEBUG:teuthology.task.selinux:Excluding smithi014: OS 'ubuntu' does not support SELinux 2020-07-13T06:06:25.333 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-07-13T06:06:25.333 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2020-07-13T06:06:25.334 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-07-13T06:06:25.334 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-07-13T06:06:25.346 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2020-07-13T06:06:25.356 INFO:teuthology.repo_utils:Fetching master from origin 2020-07-13T06:06:25.413 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-07-13T06:06:25.448 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-07-13T06:06:25.451 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi017.front.sepia.ceph.com,smithi014.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-07-13T06:06:25.844 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-07-13T06:06:26.874 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-07-13T06:06:29.843 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-07-13T06:06:29.867 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:29 +0000 (0:00:00.084) 0:00:00.084 *********** 2020-07-13T06:06:29.889 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:29 +0000 (0:00:00.021) 0:00:00.105 *********** 2020-07-13T06:06:31.738 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi017.front.sepia.ceph.com] 2020-07-13T06:06:31.852 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:31 +0000 (0:00:01.962) 0:00:02.068 *********** 2020-07-13T06:06:31.864 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] 2020-07-13T06:06:31.946 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:31 +0000 (0:00:00.091) 0:00:02.160 *********** 2020-07-13T06:06:32.291 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-07-13T06:06:32.293 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-07-13T06:06:32.327 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:32 +0000 (0:00:00.382) 0:00:02.543 *********** 2020-07-13T06:06:32.336 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-07-13T06:06:32.379 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:32 +0000 (0:00:00.051) 0:00:02.595 *********** 2020-07-13T06:06:32.969 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-07-13T06:06:32.971 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} changed: [smithi014.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-07-13T06:06:33.095 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:33 +0000 (0:00:00.716) 0:00:03.312 *********** 2020-07-13T06:06:33.133 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:33 +0000 (0:00:00.037) 0:00:03.349 *********** 2020-07-13T06:06:33.439 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** changed: [smithi014.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.059453", "end": "2020-07-13 06:06:33.629285", "rc": 0, "start": "2020-07-13 06:06:33.569832", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-07-13T06:06:33.479 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:33 +0000 (0:00:00.346) 0:00:03.695 *********** 2020-07-13T06:06:33.488 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.074884", "end": "2020-07-13 06:06:32.809988", "rc": 0, "start": "2020-07-13 06:06:32.735104", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-07-13T06:06:33.527 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:33 +0000 (0:00:00.048) 0:00:03.744 *********** 2020-07-13T06:06:33.756 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-07-13T06:06:33.758 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-13T06:06:33.810 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:33 +0000 (0:00:00.282) 0:00:04.026 *********** 2020-07-13T06:06:33.828 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-13T06:06:33.876 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:33 +0000 (0:00:00.065) 0:00:04.092 *********** 2020-07-13T06:06:34.596 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-07-13T06:06:34.599 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1594620393.893917-281268210387122/source", "state": "file", "uid": 0} 2020-07-13T06:06:34.642 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:34 +0000 (0:00:00.764) 0:00:04.857 *********** 2020-07-13T06:06:34.653 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1594620393.8375561-83390810446932/source", "state": "file", "uid": 0} 2020-07-13T06:06:34.694 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:34 +0000 (0:00:00.050) 0:00:04.908 *********** 2020-07-13T06:06:34.996 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtC 2020-07-13T06:06:34.998 INFO:teuthology.task.ansible.out:toHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1 2020-07-13T06:06:35.000 INFO:teuthology.task.ansible.out:cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P 2020-07-13T06:06:35.002 INFO:teuthology.task.ansible.out:2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-07-13T06:06:35.145 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-07-13T06:06:35.153 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-07-13T06:06:35.307 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:35 +0000 (0:00:00.614) 0:00:05.523 *********** 2020-07-13T06:06:35.343 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:35 +0000 (0:00:00.036) 0:00:05.560 *********** 2020-07-13T06:06:36.070 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi014.front.sepia.ceph.com] 2020-07-13T06:06:36.129 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] 2020-07-13T06:06:36.181 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-07-13T06:06:36.183 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:36 +0000 (0:00:00.837) 0:00:06.397 *********** 2020-07-13T06:06:36.259 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:36 +0000 (0:00:00.077) 0:00:06.475 *********** 2020-07-13T06:06:36.603 INFO:teuthology.task.ansible.out: 2020-07-13T06:06:36.606 INFO:teuthology.task.ansible.out:TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi014.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.170779", "end": "2020-07-13 06:06:36.568426", "rc": 0, "start": "2020-07-13 06:06:36.397647", "stderr": "", "stderr_lines": [], "stdout": "8cda0c31b17b2a5b38e6a918a90e759c14477f26", "stdout_lines": ["8cda0c31b17b2a5b38e6a918a90e759c14477f26"]} 2020-07-13T06:06:36.657 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:36 +0000 (0:00:00.398) 0:00:06.873 *********** 2020-07-13T06:06:36.659 INFO:teuthology.task.ansible.out: 2020-07-13T06:06:36.699 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.163497", "end": "2020-07-13 06:06:36.664668", "rc": 0, "start": "2020-07-13 06:06:36.501171", "stderr": "", "stderr_lines": [], "stdout": "8cda0c31b17b2a5b38e6a918a90e759c14477f26", "stdout_lines": ["8cda0c31b17b2a5b38e6a918a90e759c14477f26"]} 2020-07-13T06:06:36.746 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:36 +0000 (0:00:00.084) 0:00:06.957 *********** 2020-07-13T06:06:36.888 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-07-13T06:06:36.890 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.008631", "end": "2020-07-13 06:06:36.253903", "failed_when_result": false, "rc": 0, "start": "2020-07-13 06:06:36.245272", "stderr": "", "stderr_lines": [], "stdout": "8cda0c31b17b2a5b38e6a918a90e759c14477f26", "stdout_lines": ["8cda0c31b17b2a5b38e6a918a90e759c14477f26"]} 2020-07-13T06:06:36.927 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:36 +0000 (0:00:00.186) 0:00:07.144 *********** 2020-07-13T06:06:36.973 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-07-13T06:06:37.006 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:37 +0000 (0:00:00.078) 0:00:07.222 *********** 2020-07-13T06:06:37.016 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-07-13T06:06:37.018 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.167577", "end": "2020-07-13 06:06:37.193069", "failed_when_result": false, "rc": 0, "start": "2020-07-13 06:06:37.025492", "stderr": "", "stderr_lines": [], "stdout": "8cda0c31b17b2a5b38e6a918a90e759c14477f26", "stdout_lines": ["8cda0c31b17b2a5b38e6a918a90e759c14477f26"]} 2020-07-13T06:06:37.056 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:37 +0000 (0:00:00.049) 0:00:07.272 *********** 2020-07-13T06:06:37.069 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-07-13T06:06:37.071 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.112 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:37 +0000 (0:00:00.056) 0:00:07.328 *********** 2020-07-13T06:06:37.123 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** ok: [smithi014.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-07-13T06:06:37.163 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:37 +0000 (0:00:00.051) 0:00:07.380 *********** 2020-07-13T06:06:37.173 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-07-13T06:06:37.174 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.213 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:37 +0000 (0:00:00.049) 0:00:07.429 *********** 2020-07-13T06:06:37.222 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-07-13T06:06:37.223 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.261 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:37 +0000 (0:00:00.047) 0:00:07.477 *********** 2020-07-13T06:06:37.270 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-07-13T06:06:37.271 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.311 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:37 +0000 (0:00:00.050) 0:00:07.528 *********** 2020-07-13T06:06:37.323 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.361 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:37 +0000 (0:00:00.049) 0:00:07.578 *********** 2020-07-13T06:06:37.371 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-07-13T06:06:37.373 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.409 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:37 +0000 (0:00:00.047) 0:00:07.626 *********** 2020-07-13T06:06:37.427 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.473 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:37 +0000 (0:00:00.063) 0:00:07.690 *********** 2020-07-13T06:06:37.485 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-07-13T06:06:37.487 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.527 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:37 +0000 (0:00:00.053) 0:00:07.744 *********** 2020-07-13T06:06:37.537 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-07-13T06:06:37.539 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.579 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:37 +0000 (0:00:00.051) 0:00:07.795 *********** 2020-07-13T06:06:37.599 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-07-13T06:06:37.600 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.611 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.613 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.654 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:37 +0000 (0:00:00.075) 0:00:07.870 *********** 2020-07-13T06:06:37.665 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.666 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.666 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.667 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281a 2020-07-13T06:06:37.668 INFO:teuthology.task.ansible.out:o7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM1 2020-07-13T06:06:37.670 INFO:teuthology.task.ansible.out:3MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.680 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.682 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "ite 2020-07-13T06:06:37.684 INFO:teuthology.task.ansible.out:m", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.694 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.706 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.709 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.718 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.720 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilk 2020-07-13T06:06:37.721 INFO:teuthology.task.ansible.out:in@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) 2020-07-13T06:06:37.723 INFO:teuthology.task.ansible.out: => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.732 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.734 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.736 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.738 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.740 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.742 INFO:teuthology.task.ansible.out: skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.752 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.754 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.755 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.757 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.768 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.770 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.772 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.774 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.775 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.777 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.788 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.789 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.798 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "ite 2020-07-13T06:06:37.800 INFO:teuthology.task.ansible.out:m", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.802 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.804 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.819 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.821 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.823 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.825 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.833 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.835 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.835 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.836 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'} 2020-07-13T06:06:37.837 INFO:teuthology.task.ansible.out:) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a7986 2020-07-13T06:06:37.839 INFO:teuthology.task.ansible.out:6838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.848 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.850 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.851 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.852 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.865 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.866 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.878 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.880 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.890 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.893 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtD 2020-07-13T06:06:37.895 INFO:teuthology.task.ansible.out:Pj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.905 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.908 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.908 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.911 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.913 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.916 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.927 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.930 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.942 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.953 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.955 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.966 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.967 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.969 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.974 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.975 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.982 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.985 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.996 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:37.998 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.009 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.011 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.012 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.014 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.027 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", 2020-07-13T06:06:38.030 INFO:teuthology.task.ansible.out:"ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.038 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.040 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.040 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.041 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.053 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.054 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.055 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.055 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": " 2020-07-13T06:06:38.057 INFO:teuthology.task.ansible.out:swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.067 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.069 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.084 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.086 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.087 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.098 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.100 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.110 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.112 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.113 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "lau 2020-07-13T06:06:38.114 INFO:teuthology.task.ansible.out:ra@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.124 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.126 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.126 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name" 2020-07-13T06:06:38.128 INFO:teuthology.task.ansible.out:: "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.137 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.139 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.149 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.151 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.161 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.163 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.174 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.179 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.191 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.194 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, 2020-07-13T06:06:38.197 INFO:teuthology.task.ansible.out: "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 't 2020-07-13T06:06:38.199 INFO:teuthology.task.ansible.out:oabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.206 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.208 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.209 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@ 2020-07-13T06:06:38.211 INFO:teuthology.task.ansible.out:jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.219 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.220 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.222 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.224 INFO:teuthology.task.ansible.out: skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.244 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.247 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.248 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat 2020-07-13T06:06:38.261 INFO:teuthology.task.ansible.out:@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76Xx 2020-07-13T06:06:38.265 INFO:teuthology.task.ansible.out:Y0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "s 2020-07-13T06:06:38.275 INFO:teuthology.task.ansible.out:kip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.294 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.296 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph. 2020-07-13T06:06:38.298 INFO:teuthology.task.ansible.out:com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.331 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.332 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.333 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khir 2020-07-13T06:06:38.334 INFO:teuthology.task.ansible.out:emat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.375 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:38 +0000 (0:00:00.720) 0:00:08.591 *********** 2020-07-13T06:06:38.388 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.389 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.392 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.393 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.394 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": 2020-07-13T06:06:38.396 INFO:teuthology.task.ansible.out: "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.411 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-07-13T06:06:38.448 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:38 +0000 (0:00:00.073) 0:00:08.665 *********** 2020-07-13T06:06:38.493 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:38 +0000 (0:00:00.043) 0:00:08.709 *********** 2020-07-13T06:06:38.555 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:38 +0000 (0:00:00.059) 0:00:08.768 *********** 2020-07-13T06:06:38.565 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-07-13T06:06:38.570 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.610 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:38 +0000 (0:00:00.057) 0:00:08.825 *********** 2020-07-13T06:06:38.620 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.665 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:38 +0000 (0:00:00.055) 0:00:08.881 *********** 2020-07-13T06:06:38.674 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-07-13T06:06:38.676 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.712 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:38 +0000 (0:00:00.047) 0:00:08.929 *********** 2020-07-13T06:06:38.722 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.762 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:38 +0000 (0:00:00.049) 0:00:08.979 *********** 2020-07-13T06:06:38.773 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.811 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:38 +0000 (0:00:00.048) 0:00:09.027 *********** 2020-07-13T06:06:38.821 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.875 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:38 +0000 (0:00:00.062) 0:00:09.090 *********** 2020-07-13T06:06:38.891 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.928 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:38 +0000 (0:00:00.054) 0:00:09.144 *********** 2020-07-13T06:06:38.940 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:38.979 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:38 +0000 (0:00:00.050) 0:00:09.195 *********** 2020-07-13T06:06:38.991 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-07-13T06:06:38.992 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.030 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:39 +0000 (0:00:00.050) 0:00:09.246 *********** 2020-07-13T06:06:39.039 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.078 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:39 +0000 (0:00:00.048) 0:00:09.295 *********** 2020-07-13T06:06:39.088 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-07-13T06:06:39.089 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.090 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.090 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.091 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.106 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.121 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.127 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.129 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.139 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.140 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.150 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.153 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.162 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.163 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.173 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.175 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.184 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.186 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.196 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.198 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.199 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.200 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.200 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.201 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.201 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.212 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.214 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.223 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.225 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.225 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.226 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.226 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.227 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.227 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.239 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.240 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.240 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.241 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.241 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.241 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.268 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.286 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.298 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.301 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.303 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.351 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:39 +0000 (0:00:00.272) 0:00:09.567 *********** 2020-07-13T06:06:39.360 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.362 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.363 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.363 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:39.404 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:39 +0000 (0:00:00.052) 0:00:09.620 *********** 2020-07-13T06:06:40.038 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-07-13T06:06:40.041 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "checksum": "18a591c5a08bc1931a9d89d8a4fed752874b77c7", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2020-07-13T06:06:40.079 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.672) 0:00:10.292 *********** 2020-07-13T06:06:40.088 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "18a591c5a08bc1931a9d89d8a4fed752874b77c7", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2020-07-13T06:06:40.126 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.050) 0:00:10.343 *********** 2020-07-13T06:06:40.142 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi014.front.sepia.ceph.com] => { "msg": "Host smithi014.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-07-13T06:06:40.185 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.058) 0:00:10.401 *********** 2020-07-13T06:06:40.195 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => { "msg": "Host smithi017.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-07-13T06:06:40.233 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.048) 0:00:10.449 *********** 2020-07-13T06:06:40.244 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-07-13T06:06:40.246 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-07-13T06:06:40.293 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.060) 0:00:10.509 *********** 2020-07-13T06:06:40.303 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-07-13T06:06:40.344 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.050) 0:00:10.560 *********** 2020-07-13T06:06:40.353 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-07-13T06:06:40.355 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:40.396 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.052) 0:00:10.612 *********** 2020-07-13T06:06:40.406 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:40.444 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.047) 0:00:10.660 *********** 2020-07-13T06:06:40.452 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-07-13T06:06:40.454 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:40.491 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.047) 0:00:10.707 *********** 2020-07-13T06:06:40.500 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:40.546 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.047) 0:00:10.754 *********** 2020-07-13T06:06:40.554 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-07-13T06:06:40.559 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:40.598 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.056) 0:00:10.811 *********** 2020-07-13T06:06:40.605 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:40.642 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.048) 0:00:10.859 *********** 2020-07-13T06:06:40.651 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-07-13T06:06:40.653 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:40.691 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.047) 0:00:10.906 *********** 2020-07-13T06:06:40.700 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:40.749 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.058) 0:00:10.965 *********** 2020-07-13T06:06:40.760 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-07-13T06:06:40.762 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:40.797 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.048) 0:00:11.014 *********** 2020-07-13T06:06:40.806 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:40.844 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.046) 0:00:11.061 *********** 2020-07-13T06:06:40.856 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:40.894 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.049) 0:00:11.110 *********** 2020-07-13T06:06:40.903 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:40.950 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.047) 0:00:11.158 *********** 2020-07-13T06:06:40.959 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:40.995 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:40 +0000 (0:00:00.054) 0:00:11.212 *********** 2020-07-13T06:06:41.010 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:41.043 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.048) 0:00:11.260 *********** 2020-07-13T06:06:41.053 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:41.093 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.049) 0:00:11.309 *********** 2020-07-13T06:06:41.112 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:41.148 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.051) 0:00:11.360 *********** 2020-07-13T06:06:41.159 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-07-13T06:06:41.163 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi014.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-13T06:06:41.202 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.058) 0:00:11.418 *********** 2020-07-13T06:06:41.212 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-13T06:06:41.215 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-13T06:06:41.264 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.061) 0:00:11.480 *********** 2020-07-13T06:06:41.274 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-07-13T06:06:41.276 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:41.328 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.063) 0:00:11.544 *********** 2020-07-13T06:06:41.338 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:41.388 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.059) 0:00:11.604 *********** 2020-07-13T06:06:41.399 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-07-13T06:06:41.401 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:41.439 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.051) 0:00:11.655 *********** 2020-07-13T06:06:41.449 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:41.499 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.058) 0:00:11.714 *********** 2020-07-13T06:06:41.507 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-07-13T06:06:41.509 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-13T06:06:41.545 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.047) 0:00:11.761 *********** 2020-07-13T06:06:41.554 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-13T06:06:41.593 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.048) 0:00:11.809 *********** 2020-07-13T06:06:41.602 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-07-13T06:06:41.604 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:41.644 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.050) 0:00:11.860 *********** 2020-07-13T06:06:41.653 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:41.691 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.046) 0:00:11.907 *********** 2020-07-13T06:06:41.701 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-07-13T06:06:41.710 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-13T06:06:41.739 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.048) 0:00:11.955 *********** 2020-07-13T06:06:41.749 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-13T06:06:41.787 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.048) 0:00:12.003 *********** 2020-07-13T06:06:41.797 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:41.836 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.048) 0:00:12.052 *********** 2020-07-13T06:06:41.855 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:41.884 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.048) 0:00:12.101 *********** 2020-07-13T06:06:41.898 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:41.936 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.051) 0:00:12.153 *********** 2020-07-13T06:06:41.949 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:41.985 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:41 +0000 (0:00:00.049) 0:00:12.202 *********** 2020-07-13T06:06:41.995 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:42.035 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:42 +0000 (0:00:00.049) 0:00:12.252 *********** 2020-07-13T06:06:42.050 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:42.087 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:42 +0000 (0:00:00.051) 0:00:12.303 *********** 2020-07-13T06:06:42.096 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-07-13T06:06:42.099 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:42.136 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:42 +0000 (0:00:00.049) 0:00:12.353 *********** 2020-07-13T06:06:42.148 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:42.190 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:42 +0000 (0:00:00.053) 0:00:12.406 *********** 2020-07-13T06:06:42.201 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi014.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-13T06:06:42.240 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:42 +0000 (0:00:00.049) 0:00:12.455 *********** 2020-07-13T06:06:42.250 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-13T06:06:42.288 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:42 +0000 (0:00:00.048) 0:00:12.504 *********** 2020-07-13T06:06:42.297 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-07-13T06:06:42.298 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:42.349 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:42 +0000 (0:00:00.061) 0:00:12.565 *********** 2020-07-13T06:06:42.363 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:42.431 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:42 +0000 (0:00:00.081) 0:00:12.647 *********** 2020-07-13T06:06:42.444 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-07-13T06:06:42.447 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:42.485 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:42 +0000 (0:00:00.054) 0:00:12.701 *********** 2020-07-13T06:06:42.495 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:42.537 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:42 +0000 (0:00:00.051) 0:00:12.753 *********** 2020-07-13T06:06:42.554 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:42.598 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:42 +0000 (0:00:00.060) 0:00:12.814 *********** 2020-07-13T06:06:42.611 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:42.652 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:42 +0000 (0:00:00.054) 0:00:12.868 *********** 2020-07-13T06:06:42.661 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-07-13T06:06:42.663 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:42.701 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:42 +0000 (0:00:00.048) 0:00:12.917 *********** 2020-07-13T06:06:42.721 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:42.754 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:42 +0000 (0:00:00.049) 0:00:12.967 *********** 2020-07-13T06:06:42.784 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:42.812 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:42 +0000 (0:00:00.061) 0:00:13.029 *********** 2020-07-13T06:06:42.823 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:42.861 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:42 +0000 (0:00:00.049) 0:00:13.078 *********** 2020-07-13T06:06:42.872 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-07-13T06:06:42.874 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:42.919 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:42 +0000 (0:00:00.057) 0:00:13.135 *********** 2020-07-13T06:06:42.931 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:42.969 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:42 +0000 (0:00:00.048) 0:00:13.183 *********** 2020-07-13T06:06:42.977 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-07-13T06:06:42.980 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-13T06:06:43.023 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:43 +0000 (0:00:00.055) 0:00:13.239 *********** 2020-07-13T06:06:43.039 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-13T06:06:43.074 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:43 +0000 (0:00:00.050) 0:00:13.290 *********** 2020-07-13T06:06:43.083 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-07-13T06:06:43.085 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:43.127 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:43 +0000 (0:00:00.052) 0:00:13.343 *********** 2020-07-13T06:06:43.145 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:43.179 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:43 +0000 (0:00:00.052) 0:00:13.396 *********** 2020-07-13T06:06:43.190 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-07-13T06:06:43.198 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:43.229 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:43 +0000 (0:00:00.049) 0:00:13.445 *********** 2020-07-13T06:06:43.240 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:43.281 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:43 +0000 (0:00:00.051) 0:00:13.497 *********** 2020-07-13T06:06:43.289 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-07-13T06:06:43.291 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:43.342 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:43 +0000 (0:00:00.057) 0:00:13.554 *********** 2020-07-13T06:06:43.348 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:43.388 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:43 +0000 (0:00:00.049) 0:00:13.604 *********** 2020-07-13T06:06:43.413 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-07-13T06:06:43.415 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:43.426 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:43.466 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:43 +0000 (0:00:00.078) 0:00:13.682 *********** 2020-07-13T06:06:43.481 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:43.523 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:43 +0000 (0:00:00.056) 0:00:13.739 *********** 2020-07-13T06:06:43.533 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-07-13T06:06:43.534 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:43.574 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:43 +0000 (0:00:00.051) 0:00:13.790 *********** 2020-07-13T06:06:43.590 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:43.625 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:43 +0000 (0:00:00.050) 0:00:13.841 *********** 2020-07-13T06:06:43.635 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-07-13T06:06:43.637 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:43.673 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:43 +0000 (0:00:00.047) 0:00:13.889 *********** 2020-07-13T06:06:43.682 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:43.723 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:43 +0000 (0:00:00.050) 0:00:13.939 *********** 2020-07-13T06:06:43.834 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-07-13T06:06:43.836 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.013072", "end": "2020-07-13 06:06:44.027423", "rc": 0, "start": "2020-07-13 06:06:44.014351", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-07-13T06:06:43.872 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:43 +0000 (0:00:00.149) 0:00:14.089 *********** 2020-07-13T06:06:43.893 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.013415", "end": "2020-07-13 06:06:43.246431", "rc": 0, "start": "2020-07-13 06:06:43.233016", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-07-13T06:06:43.939 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:43 +0000 (0:00:00.066) 0:00:14.155 *********** 2020-07-13T06:06:43.958 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-07-13T06:06:43.960 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:44.023 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:44 +0000 (0:00:00.083) 0:00:14.239 *********** 2020-07-13T06:06:44.034 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:44.074 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:44 +0000 (0:00:00.051) 0:00:14.290 *********** 2020-07-13T06:06:44.082 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-07-13T06:06:44.084 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:44.122 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:44 +0000 (0:00:00.048) 0:00:14.338 *********** 2020-07-13T06:06:44.132 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:44.176 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:44 +0000 (0:00:00.047) 0:00:14.386 *********** 2020-07-13T06:06:44.184 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:44.220 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:44 +0000 (0:00:00.049) 0:00:14.436 *********** 2020-07-13T06:06:44.230 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:44.281 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:44 +0000 (0:00:00.060) 0:00:14.497 *********** 2020-07-13T06:06:44.293 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-07-13T06:06:44.295 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:44.338 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:44 +0000 (0:00:00.057) 0:00:14.554 *********** 2020-07-13T06:06:44.350 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:44.387 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:44 +0000 (0:00:00.049) 0:00:14.604 *********** 2020-07-13T06:06:44.397 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-07-13T06:06:44.399 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:44.444 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:44 +0000 (0:00:00.057) 0:00:14.661 *********** 2020-07-13T06:06:44.454 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:44.493 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:44 +0000 (0:00:00.047) 0:00:14.709 *********** 2020-07-13T06:06:44.502 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-07-13T06:06:44.505 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:44.542 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:44 +0000 (0:00:00.049) 0:00:14.758 *********** 2020-07-13T06:06:44.552 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:44.590 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:44 +0000 (0:00:00.048) 0:00:14.807 *********** 2020-07-13T06:06:44.600 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-07-13T06:06:44.606 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:44.639 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:44 +0000 (0:00:00.049) 0:00:14.856 *********** 2020-07-13T06:06:44.648 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:44.687 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:44 +0000 (0:00:00.047) 0:00:14.903 *********** 2020-07-13T06:06:45.142 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-07-13T06:06:45.144 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.303241", "end": "2020-07-13 06:06:44.500462", "rc": 0, "start": "2020-07-13 06:06:44.197221", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:06:45.180 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:45 +0000 (0:00:00.492) 0:00:15.396 *********** 2020-07-13T06:06:45.212 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-07-13T06:06:45.214 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:45.265 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:45 +0000 (0:00:00.084) 0:00:15.481 *********** 2020-07-13T06:06:45.307 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-07-13T06:06:45.309 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.537612", "end": "2020-07-13 06:06:45.500337", "rc": 0, "start": "2020-07-13 06:06:44.962725", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:06:45.345 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:45 +0000 (0:00:00.080) 0:00:15.561 *********** 2020-07-13T06:06:45.376 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:45.414 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:45 +0000 (0:00:00.069) 0:00:15.631 *********** 2020-07-13T06:06:45.436 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-07-13T06:06:45.438 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.023202", "end": "2020-07-13 06:06:44.797314", "rc": 0, "start": "2020-07-13 06:06:44.774112", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:06:45.484 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:45 +0000 (0:00:00.068) 0:00:15.700 *********** 2020-07-13T06:06:45.604 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.023549", "end": "2020-07-13 06:06:45.792490", "rc": 0, "start": "2020-07-13 06:06:45.768941", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:06:45.654 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:45 +0000 (0:00:00.163) 0:00:15.864 *********** 2020-07-13T06:06:48.677 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** ok: [smithi017.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1594620377, "cache_updated": false, "changed": false} 2020-07-13T06:06:48.724 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:48 +0000 (0:00:03.076) 0:00:18.940 *********** 2020-07-13T06:06:48.749 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1594620381, "cache_updated": false, "changed": false} 2020-07-13T06:06:48.792 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:48 +0000 (0:00:00.067) 0:00:19.008 *********** 2020-07-13T06:06:49.602 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-07-13T06:06:49.604 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1594620377, "cache_updated": false, "changed": false} 2020-07-13T06:06:49.641 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:49 +0000 (0:00:00.849) 0:00:19.857 *********** 2020-07-13T06:06:49.672 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:49.711 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:49 +0000 (0:00:00.069) 0:00:19.927 *********** 2020-07-13T06:06:49.747 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** ok: [smithi014.front.sepia.ceph.com] => {"cache_update_time": 1594620381, "cache_updated": false, "changed": false} 2020-07-13T06:06:49.781 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:49 +0000 (0:00:00.069) 0:00:19.997 *********** 2020-07-13T06:06:49.811 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-07-13T06:06:49.813 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:49.851 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:49 +0000 (0:00:00.070) 0:00:20.068 *********** 2020-07-13T06:06:50.067 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-07-13T06:06:50.113 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:50 +0000 (0:00:00.261) 0:00:20.329 *********** 2020-07-13T06:06:50.242 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-07-13T06:06:50.296 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:50 +0000 (0:00:00.182) 0:00:20.512 *********** 2020-07-13T06:06:50.979 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1594620377, "cache_updated": false, "changed": false} 2020-07-13T06:06:51.044 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:51 +0000 (0:00:00.748) 0:00:21.261 *********** 2020-07-13T06:06:51.168 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"cache_update_time": 1594620381, "cache_updated": false, "changed": false} 2020-07-13T06:06:51.225 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:51 +0000 (0:00:00.180) 0:00:21.441 *********** 2020-07-13T06:06:51.453 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi017.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-07-13T06:06:51.654 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-07-13T06:06:51.819 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-07-13T06:06:51.993 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-07-13T06:06:52.121 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-07-13T06:06:52.186 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:52 +0000 (0:00:00.961) 0:00:22.403 *********** 2020-07-13T06:06:52.341 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-07-13T06:06:52.345 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-07-13T06:06:52.347 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-07-13T06:06:52.391 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:52 +0000 (0:00:00.204) 0:00:22.607 *********** 2020-07-13T06:06:52.441 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:52 +0000 (0:00:00.049) 0:00:22.657 *********** 2020-07-13T06:06:52.537 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-07-13T06:06:52.591 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:52 +0000 (0:00:00.142) 0:00:22.800 *********** 2020-07-13T06:06:52.830 INFO:teuthology.task.ansible.out: 2020-07-13T06:06:52.833 INFO:teuthology.task.ansible.out:TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi017.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-07-13T06:06:53.214 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-07-13T06:06:53.237 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-07-13T06:06:53.550 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-07-13T06:06:53.618 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:53 +0000 (0:00:01.032) 0:00:23.832 *********** 2020-07-13T06:06:53.625 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-07-13T06:06:53.670 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-07-13T06:06:53.673 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-07-13T06:06:53.707 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:53 +0000 (0:00:00.091) 0:00:23.924 *********** 2020-07-13T06:06:53.738 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-07-13T06:06:53.740 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:53.777 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:53 +0000 (0:00:00.069) 0:00:23.993 *********** 2020-07-13T06:06:53.808 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* skipping: [smithi017.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-13T06:06:53.864 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:53 +0000 (0:00:00.086) 0:00:24.080 *********** 2020-07-13T06:06:53.919 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-07-13T06:06:53.922 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:53.931 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:53.933 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:53.934 INFO:teuthology.task.ansible.out: 2020-07-13T06:06:53.935 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:53.937 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi014.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-07-13T06:06:53.984 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:53 +0000 (0:00:00.120) 0:00:24.200 *********** 2020-07-13T06:06:54.034 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:54 +0000 (0:00:00.049) 0:00:24.249 *********** 2020-07-13T06:06:54.043 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-07-13T06:06:54.045 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-07-13T06:06:54.081 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:54 +0000 (0:00:00.047) 0:00:24.297 *********** 2020-07-13T06:06:54.092 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-07-13T06:06:54.094 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:54.130 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:54 +0000 (0:00:00.048) 0:00:24.345 *********** 2020-07-13T06:06:54.138 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-07-13T06:06:54.142 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:54.179 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:54 +0000 (0:00:00.049) 0:00:24.395 *********** 2020-07-13T06:06:54.223 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-07-13T06:06:54.226 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-13T06:06:54.286 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:54 +0000 (0:00:00.100) 0:00:24.495 *********** 2020-07-13T06:06:54.351 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-07-13T06:06:54.353 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:54.363 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:54.364 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:54.374 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-07-13T06:06:54.414 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:54 +0000 (0:00:00.134) 0:00:24.630 *********** 2020-07-13T06:06:54.458 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-07-13T06:06:54.460 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:54.497 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:54 +0000 (0:00:00.082) 0:00:24.713 *********** 2020-07-13T06:06:55.063 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-07-13T06:06:55.064 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1594620377, "cache_updated": false, "changed": false} 2020-07-13T06:06:55.107 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:55 +0000 (0:00:00.610) 0:00:25.323 *********** 2020-07-13T06:06:55.391 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"cache_update_time": 1594620381, "cache_updated": false, "changed": false} 2020-07-13T06:06:55.467 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:55 +0000 (0:00:00.360) 0:00:25.683 *********** 2020-07-13T06:06:55.478 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-07-13T06:06:55.479 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-07-13T06:06:55.518 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:55 +0000 (0:00:00.050) 0:00:25.734 *********** 2020-07-13T06:06:55.855 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-07-13T06:06:55.888 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:55 +0000 (0:00:00.370) 0:00:26.104 *********** 2020-07-13T06:06:55.900 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-07-13T06:06:55.902 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-07-13T06:06:55.947 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:55 +0000 (0:00:00.058) 0:00:26.163 *********** 2020-07-13T06:06:56.133 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ ok: [smithi017.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-07-13T06:06:56.218 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-07-13T06:06:56.220 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-07-13T06:06:56.264 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:56 +0000 (0:00:00.317) 0:00:26.481 *********** 2020-07-13T06:06:56.279 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-07-13T06:06:56.280 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-07-13T06:06:56.332 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:56 +0000 (0:00:00.066) 0:00:26.547 *********** 2020-07-13T06:06:56.360 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-07-13T06:06:56.362 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:56.398 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:56 +0000 (0:00:00.065) 0:00:26.613 *********** 2020-07-13T06:06:56.441 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-07-13T06:06:56.443 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-07-13T06:06:56.453 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-07-13T06:06:56.454 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-13T06:06:56.493 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:56 +0000 (0:00:00.091) 0:00:26.705 *********** 2020-07-13T06:06:56.529 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-07-13T06:06:56.530 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:56.566 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:56 +0000 (0:00:00.077) 0:00:26.783 *********** 2020-07-13T06:06:56.577 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-07-13T06:06:56.578 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-07-13T06:06:56.623 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:56 +0000 (0:00:00.056) 0:00:26.839 *********** 2020-07-13T06:06:56.663 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:56.695 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:56 +0000 (0:00:00.072) 0:00:26.912 *********** 2020-07-13T06:06:56.734 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-07-13T06:06:56.736 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-13T06:06:56.780 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:56 +0000 (0:00:00.084) 0:00:26.996 *********** 2020-07-13T06:06:56.830 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:56.868 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:56 +0000 (0:00:00.086) 0:00:27.083 *********** 2020-07-13T06:06:56.930 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-07-13T06:06:56.934 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "9c3d1d79583cf76a6d81bb2ec150bc4af5ca3c5c", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1166, "state": "file", "uid": 0} 2020-07-13T06:06:56.975 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:56 +0000 (0:00:00.108) 0:00:27.191 *********** 2020-07-13T06:06:57.251 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "checksum": "9c3d1d79583cf76a6d81bb2ec150bc4af5ca3c5c", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1166, "state": "file", "uid": 0} 2020-07-13T06:06:57.310 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:57 +0000 (0:00:00.335) 0:00:27.527 *********** 2020-07-13T06:06:57.731 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-07-13T06:06:57.733 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-07-13 06:05:39 UTC", "ActiveEnterTimestampMonotonic": "17548415", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target local-fs.target sysinit.target -.mount system.slice systemd-tmpfiles-setup.service nss-lookup.target var-run.mount systemd-journald.socket remote-fs.target basic.target time-sync.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-07-13 06:05:39 UTC", "AssertTimestampMonotonic": "17547597", "Before": "getty@tty1.service xdm.service multi-user.target shutdown.target plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-07-13 06:05:39 UTC", "ConditionTimestampMonotonic": "17547597", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-07-13T06:06:57.734 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1103", "ExecMainStartTimestamp": "Mon 2020-07-13 06:05:39 UTC", "ExecMainStartTimestampMonotonic": "17548384", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-07-13 06:05:39 UTC", "InactiveExitTimestampMonotonic": "17548415", "InvocationID": "9b39855cd0c3458881a44114ef9a6197", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2020-07-13T06:06:57.736 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1103", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2020-07-13T06:06:57.738 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-07-13 06:05:39 UTC", "StateChangeTimestampMonotonic": "17548415", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-07-13 06:05:39 UTC", "WatchdogTimestampMonotonic": "17548414", "WatchdogUSec": "0"}} ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-07-13 06:05:36 UTC", "ActiveEnterTimestampMonotonic": "18210503", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target local-fs.target basic.target var-run.mount nss-lookup.target systemd-tmpfiles-setup.service -.mount system.slice systemd-journald.socket sysinit.target r 2020-07-13T06:06:57.743 INFO:teuthology.task.ansible.out:emote-fs.target network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-07-13 06:05:36 UTC", "AssertTimestampMonotonic": "18209750", "Before": "plymouth-quit.service shutdown.target multi-user.target xdm.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-07-13 06:05:36 UTC", "ConditionTimestampMonotonic": "18209750", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-server.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1058", "ExecMainStartTimestamp": "Mon 2020-07-13 06:05:36 UTC", "ExecMainStartTimestampMonotonic": 2020-07-13T06:06:57.745 INFO:teuthology.task.ansible.out:"18210472", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-07-13 06:05:36 UTC", "InactiveExitTimestampMonotonic": "18210503", "InvocationID": "312b26b8f8414dd9a35b67429a51de94", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "102 2020-07-13T06:06:57.747 INFO:teuthology.task.ansible.out:4", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1058", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "Star 2020-07-13T06:06:57.748 INFO:teuthology.task.ansible.out:tLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-07-13 06:05:36 UTC", "StateChangeTimestampMonotonic": "18210503", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-07-13 06:05:36 UTC", "WatchdogTimestampMonotonic": "18210502", "WatchdogUSec": "0"}} 2020-07-13T06:06:57.790 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:57 +0000 (0:00:00.473) 0:00:28.000 *********** 2020-07-13T06:06:57.827 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:57 +0000 (0:00:00.042) 0:00:28.043 *********** 2020-07-13T06:06:57.839 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:57.883 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:57 +0000 (0:00:00.055) 0:00:28.099 *********** 2020-07-13T06:06:57.896 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:57.936 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:57 +0000 (0:00:00.053) 0:00:28.152 *********** 2020-07-13T06:06:57.952 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-07-13T06:06:57.954 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:57.993 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:57 +0000 (0:00:00.057) 0:00:28.210 *********** 2020-07-13T06:06:58.003 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:58.042 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.048) 0:00:28.258 *********** 2020-07-13T06:06:58.051 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-07-13T06:06:58.053 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:58.090 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.047) 0:00:28.306 *********** 2020-07-13T06:06:58.100 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:58.140 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.050) 0:00:28.356 *********** 2020-07-13T06:06:58.154 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-07-13T06:06:58.156 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:58.194 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.054) 0:00:28.411 *********** 2020-07-13T06:06:58.207 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:58.248 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.053) 0:00:28.464 *********** 2020-07-13T06:06:58.258 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-07-13T06:06:58.263 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:58.296 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.048) 0:00:28.512 *********** 2020-07-13T06:06:58.306 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:58.345 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.049) 0:00:28.561 *********** 2020-07-13T06:06:58.363 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:58.401 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.055) 0:00:28.617 *********** 2020-07-13T06:06:58.411 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:58.450 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.048) 0:00:28.666 *********** 2020-07-13T06:06:58.459 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-07-13T06:06:58.461 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:58.503 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.053) 0:00:28.719 *********** 2020-07-13T06:06:58.513 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:58.550 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.047) 0:00:28.767 *********** 2020-07-13T06:06:58.565 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:58.600 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.049) 0:00:28.816 *********** 2020-07-13T06:06:58.610 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:58.650 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.050) 0:00:28.866 *********** 2020-07-13T06:06:58.661 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:58.700 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.050) 0:00:28.916 *********** 2020-07-13T06:06:58.710 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:58.749 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.048) 0:00:28.965 *********** 2020-07-13T06:06:58.791 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:58.808 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.059) 0:00:29.025 *********** 2020-07-13T06:06:58.819 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:58.865 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.056) 0:00:29.081 *********** 2020-07-13T06:06:58.876 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-07-13T06:06:58.878 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-07-13T06:06:58.918 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.053) 0:00:29.135 *********** 2020-07-13T06:06:58.930 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-07-13T06:06:58.979 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:58 +0000 (0:00:00.060) 0:00:29.195 *********** 2020-07-13T06:06:58.988 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-07-13T06:06:58.990 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-07-13T06:06:59.041 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:59 +0000 (0:00:00.062) 0:00:29.257 *********** 2020-07-13T06:06:59.052 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-07-13T06:06:59.093 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:59 +0000 (0:00:00.052) 0:00:29.310 *********** 2020-07-13T06:06:59.103 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-07-13T06:06:59.107 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-07-13T06:06:59.143 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:59 +0000 (0:00:00.049) 0:00:29.360 *********** 2020-07-13T06:06:59.155 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-07-13T06:06:59.204 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:59 +0000 (0:00:00.060) 0:00:29.420 *********** 2020-07-13T06:06:59.225 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-07-13T06:06:59.226 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-07-13T06:06:59.268 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:59 +0000 (0:00:00.064) 0:00:29.485 *********** 2020-07-13T06:06:59.283 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-07-13T06:06:59.334 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:59 +0000 (0:00:00.065) 0:00:29.550 *********** 2020-07-13T06:06:59.421 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-07-13T06:06:59.423 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-07-13T06:06:59.464 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:59 +0000 (0:00:00.130) 0:00:29.681 *********** 2020-07-13T06:06:59.496 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-07-13T06:06:59.535 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:59 +0000 (0:00:00.069) 0:00:29.751 *********** 2020-07-13T06:06:59.610 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-07-13T06:06:59.611 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "gid": 1222, "name": "ubuntu", "state": "present", "system": false} 2020-07-13T06:06:59.651 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:59 +0000 (0:00:00.116) 0:00:29.867 *********** 2020-07-13T06:06:59.682 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-07-13T06:06:59.685 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:59.722 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:59 +0000 (0:00:00.071) 0:00:29.939 *********** 2020-07-13T06:06:59.734 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-07-13T06:06:59.736 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "gid": 1222, "name": "ubuntu", "state": "present", "system": false} 2020-07-13T06:06:59.774 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:59 +0000 (0:00:00.051) 0:00:29.990 *********** 2020-07-13T06:06:59.789 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:59.829 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:59 +0000 (0:00:00.054) 0:00:30.045 *********** 2020-07-13T06:06:59.839 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-07-13T06:06:59.841 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:59.897 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:59 +0000 (0:00:00.067) 0:00:30.113 *********** 2020-07-13T06:06:59.930 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-07-13T06:06:59.931 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:06:59.966 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:06:59 +0000 (0:00:00.069) 0:00:30.182 *********** 2020-07-13T06:07:00.042 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi014.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-07-13T06:07:00.080 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:00 +0000 (0:00:00.113) 0:00:30.296 *********** 2020-07-13T06:07:00.142 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-07-13T06:07:00.186 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:00 +0000 (0:00:00.106) 0:00:30.402 *********** 2020-07-13T06:07:00.512 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-07-13T06:07:00.514 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-07-13T06:07:00.552 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:00 +0000 (0:00:00.365) 0:00:30.768 *********** 2020-07-13T06:07:00.594 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi014.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi014"}, "changed": false} 2020-07-13T06:07:00.634 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:00 +0000 (0:00:00.082) 0:00:30.850 *********** 2020-07-13T06:07:00.644 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-07-13T06:07:00.645 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-07-13T06:07:00.685 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:00 +0000 (0:00:00.051) 0:00:30.901 *********** 2020-07-13T06:07:00.727 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi017"}, "changed": false} 2020-07-13T06:07:00.765 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:00 +0000 (0:00:00.080) 0:00:30.982 *********** 2020-07-13T06:07:01.630 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-07-13T06:07:01.631 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi017", "ansible_hostname": "smithi017", "ansible_nodename": "smithi017"}, "changed": false, "name": "smithi017"} 2020-07-13T06:07:01.679 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:01 +0000 (0:00:00.913) 0:00:31.895 *********** 2020-07-13T06:07:01.689 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi014", "ansible_hostname": "smithi014", "ansible_nodename": "smithi014"}, "changed": false, "name": "smithi014"} 2020-07-13T06:07:01.729 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:01 +0000 (0:00:00.049) 0:00:31.945 *********** 2020-07-13T06:07:02.048 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-07-13T06:07:02.097 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:02 +0000 (0:00:00.366) 0:00:32.312 *********** 2020-07-13T06:07:02.116 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-07-13T06:07:02.160 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:02 +0000 (0:00:00.064) 0:00:32.376 *********** 2020-07-13T06:07:02.492 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-07-13T06:07:02.499 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-07-13T06:07:02.539 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:02 +0000 (0:00:00.379) 0:00:32.755 *********** 2020-07-13T06:07:02.551 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-07-13T06:07:02.598 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:02 +0000 (0:00:00.053) 0:00:32.809 *********** 2020-07-13T06:07:02.921 INFO:teuthology.task.ansible.out:FAILED - RETRYING: Add ssh pubkeys (24 retries left). 2020-07-13T06:07:02.995 INFO:teuthology.task.ansible.out:FAILED - RETRYING: Add ssh pubkeys (24 retries left). 2020-07-13T06:07:08.430 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** ok: [smithi017.front.sepia.ceph.com] => {"attempts": 2, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-07-13T06:07:08.466 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:08 +0000 (0:00:05.872) 0:00:38.682 *********** 2020-07-13T06:07:08.481 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"attempts": 2, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-07-13T06:07:08.517 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:08 +0000 (0:00:00.051) 0:00:38.733 *********** 2020-07-13T06:07:08.527 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-07-13T06:07:08.529 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:08.571 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:08 +0000 (0:00:00.053) 0:00:38.787 *********** 2020-07-13T06:07:08.581 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:08.621 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:08 +0000 (0:00:00.050) 0:00:38.837 *********** 2020-07-13T06:07:08.631 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-07-13T06:07:08.633 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:08.672 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:08 +0000 (0:00:00.050) 0:00:38.888 *********** 2020-07-13T06:07:08.682 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:08.723 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:08 +0000 (0:00:00.051) 0:00:38.939 *********** 2020-07-13T06:07:08.733 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-07-13T06:07:08.735 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:08.772 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:08 +0000 (0:00:00.049) 0:00:38.989 *********** 2020-07-13T06:07:08.792 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:08.833 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:08 +0000 (0:00:00.060) 0:00:39.049 *********** 2020-07-13T06:07:08.854 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:08.885 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:08 +0000 (0:00:00.052) 0:00:39.101 *********** 2020-07-13T06:07:08.896 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:08.936 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:08 +0000 (0:00:00.050) 0:00:39.152 *********** 2020-07-13T06:07:08.945 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-07-13T06:07:08.949 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:08.991 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:08 +0000 (0:00:00.049) 0:00:39.202 *********** 2020-07-13T06:07:08.995 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:09.037 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.050) 0:00:39.253 *********** 2020-07-13T06:07:09.046 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-07-13T06:07:09.047 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:09.086 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.049) 0:00:39.303 *********** 2020-07-13T06:07:09.096 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:09.136 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.048) 0:00:39.352 *********** 2020-07-13T06:07:09.147 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-07-13T06:07:09.149 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:09.188 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.052) 0:00:39.404 *********** 2020-07-13T06:07:09.203 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:09.240 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.051) 0:00:39.456 *********** 2020-07-13T06:07:09.254 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:09.295 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.055) 0:00:39.511 *********** 2020-07-13T06:07:09.305 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:09.346 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.050) 0:00:39.562 *********** 2020-07-13T06:07:09.358 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-07-13T06:07:09.396 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.050) 0:00:39.612 *********** 2020-07-13T06:07:09.448 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.051) 0:00:39.664 *********** 2020-07-13T06:07:09.459 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-07-13T06:07:09.499 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.051) 0:00:39.715 *********** 2020-07-13T06:07:09.551 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.051) 0:00:39.767 *********** 2020-07-13T06:07:09.564 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-07-13T06:07:09.605 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.052) 0:00:39.820 *********** 2020-07-13T06:07:09.658 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.054) 0:00:39.874 *********** 2020-07-13T06:07:09.668 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-07-13T06:07:09.670 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:09.709 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.050) 0:00:39.925 *********** 2020-07-13T06:07:09.719 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:09.765 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.052) 0:00:39.977 *********** 2020-07-13T06:07:09.773 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi017.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:09.811 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.050) 0:00:40.027 *********** 2020-07-13T06:07:09.829 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:09.837 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:09.866 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.055) 0:00:40.082 *********** 2020-07-13T06:07:09.875 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-07-13T06:07:09.882 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:09.913 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.047) 0:00:40.130 *********** 2020-07-13T06:07:09.924 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:09.963 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:09 +0000 (0:00:00.050) 0:00:40.180 *********** 2020-07-13T06:07:09.974 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-13T06:07:09.977 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.013 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.049) 0:00:40.229 *********** 2020-07-13T06:07:10.023 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.068 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.050) 0:00:40.280 *********** 2020-07-13T06:07:10.075 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.112 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.048) 0:00:40.328 *********** 2020-07-13T06:07:10.121 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.165 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.053) 0:00:40.381 *********** 2020-07-13T06:07:10.177 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.214 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.048) 0:00:40.430 *********** 2020-07-13T06:07:10.225 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.268 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.054) 0:00:40.484 *********** 2020-07-13T06:07:10.278 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-07-13T06:07:10.280 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.317 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.048) 0:00:40.533 *********** 2020-07-13T06:07:10.326 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.365 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.048) 0:00:40.582 *********** 2020-07-13T06:07:10.379 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.417 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.049) 0:00:40.632 *********** 2020-07-13T06:07:10.425 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.467 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.051) 0:00:40.683 *********** 2020-07-13T06:07:10.477 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-07-13T06:07:10.478 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.515 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.047) 0:00:40.731 *********** 2020-07-13T06:07:10.524 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.562 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.047) 0:00:40.778 *********** 2020-07-13T06:07:10.573 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-07-13T06:07:10.575 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.611 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.048) 0:00:40.827 *********** 2020-07-13T06:07:10.621 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.664 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.053) 0:00:40.881 *********** 2020-07-13T06:07:10.675 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-07-13T06:07:10.677 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.714 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.049) 0:00:40.930 *********** 2020-07-13T06:07:10.724 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.764 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.050) 0:00:40.980 *********** 2020-07-13T06:07:10.776 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-07-13T06:07:10.778 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.815 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.051) 0:00:41.031 *********** 2020-07-13T06:07:10.827 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.876 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.060) 0:00:41.091 *********** 2020-07-13T06:07:10.890 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.929 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.053) 0:00:41.145 *********** 2020-07-13T06:07:10.940 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:10.979 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:10 +0000 (0:00:00.049) 0:00:41.195 *********** 2020-07-13T06:07:10.991 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-07-13T06:07:10.993 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:11.031 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:11 +0000 (0:00:00.052) 0:00:41.247 *********** 2020-07-13T06:07:11.042 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:11.093 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:11 +0000 (0:00:00.061) 0:00:41.309 *********** 2020-07-13T06:07:11.104 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-07-13T06:07:11.106 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:11.143 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:11 +0000 (0:00:00.050) 0:00:41.360 *********** 2020-07-13T06:07:11.153 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:11.193 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:11 +0000 (0:00:00.049) 0:00:41.409 *********** 2020-07-13T06:07:11.205 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:11.244 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:11 +0000 (0:00:00.049) 0:00:41.459 *********** 2020-07-13T06:07:11.254 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:11.297 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:11 +0000 (0:00:00.054) 0:00:41.513 *********** 2020-07-13T06:07:11.309 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-07-13T06:07:11.311 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:11.348 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:11 +0000 (0:00:00.051) 0:00:41.564 *********** 2020-07-13T06:07:11.358 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:11.398 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:11 +0000 (0:00:00.049) 0:00:41.614 *********** 2020-07-13T06:07:11.408 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-07-13T06:07:11.410 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:11.450 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:11 +0000 (0:00:00.049) 0:00:41.663 *********** 2020-07-13T06:07:11.458 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:11.500 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:11 +0000 (0:00:00.052) 0:00:41.716 *********** 2020-07-13T06:07:11.510 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-07-13T06:07:11.512 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:11.550 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:11 +0000 (0:00:00.050) 0:00:41.766 *********** 2020-07-13T06:07:11.560 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:11.603 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:11 +0000 (0:00:00.052) 0:00:41.819 *********** 2020-07-13T06:07:11.616 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-07-13T06:07:11.618 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:11.656 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:11 +0000 (0:00:00.052) 0:00:41.872 *********** 2020-07-13T06:07:11.668 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:11.707 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:11 +0000 (0:00:00.051) 0:00:41.923 *********** 2020-07-13T06:07:11.718 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-07-13T06:07:11.720 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:11.761 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:11 +0000 (0:00:00.049) 0:00:41.973 *********** 2020-07-13T06:07:11.770 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:11.807 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:11 +0000 (0:00:00.050) 0:00:42.023 *********** 2020-07-13T06:07:11.818 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-07-13T06:07:11.819 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:11.861 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:11 +0000 (0:00:00.049) 0:00:42.072 *********** 2020-07-13T06:07:11.867 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:11.912 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:11 +0000 (0:00:00.055) 0:00:42.128 *********** 2020-07-13T06:07:12.008 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-07-13T06:07:12.010 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.004515", "end": "2020-07-13 06:07:11.374879", "rc": 0, "start": "2020-07-13 06:07:11.370364", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:07:12.049 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:12 +0000 (0:00:00.137) 0:00:42.266 *********** 2020-07-13T06:07:12.071 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005272", "end": "2020-07-13 06:07:12.268067", "rc": 0, "start": "2020-07-13 06:07:12.262795", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:07:12.117 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:12 +0000 (0:00:00.067) 0:00:42.333 *********** 2020-07-13T06:07:12.129 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-07-13T06:07:12.167 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:12 +0000 (0:00:00.050) 0:00:42.384 *********** 2020-07-13T06:07:12.219 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:12 +0000 (0:00:00.051) 0:00:42.435 *********** 2020-07-13T06:07:12.526 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-07-13T06:07:12.528 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-07-13T06:07:12.584 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:12 +0000 (0:00:00.361) 0:00:42.797 *********** 2020-07-13T06:07:12.617 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Set apt preferences] ****************************************** ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-07-13T06:07:12.658 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:12 +0000 (0:00:00.074) 0:00:42.872 *********** 2020-07-13T06:07:12.695 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:12 +0000 (0:00:00.038) 0:00:42.911 *********** 2020-07-13T06:07:12.714 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:12.745 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:12 +0000 (0:00:00.050) 0:00:42.961 *********** 2020-07-13T06:07:14.024 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* ok: [smithi017.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-07-13T06:07:14.057 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-07-13T06:07:15.006 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-07-13T06:07:15.038 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:15 +0000 (0:00:02.292) 0:00:45.254 *********** 2020-07-13T06:07:15.052 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-07-13T06:07:15.088 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:15 +0000 (0:00:00.050) 0:00:45.304 *********** 2020-07-13T06:07:15.954 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-07-13T06:07:15.956 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1594620377, "cache_updated": false, "changed": false} 2020-07-13T06:07:15.997 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:15 +0000 (0:00:00.908) 0:00:46.213 *********** 2020-07-13T06:07:16.024 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"cache_update_time": 1594620381, "cache_updated": false, "changed": false} 2020-07-13T06:07:16.064 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:16 +0000 (0:00:00.066) 0:00:46.280 *********** 2020-07-13T06:07:16.074 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-07-13T06:07:16.116 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:16 +0000 (0:00:00.051) 0:00:46.331 *********** 2020-07-13T06:07:16.167 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:16 +0000 (0:00:00.051) 0:00:46.383 *********** 2020-07-13T06:07:16.299 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo 2020-07-13T06:07:16.301 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.035585", "end": "2020-07-13 06:07:15.660416", "rc": 0, "start": "2020-07-13 06:07:15.624831", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:07:16.346 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:16 +0000 (0:00:00.177) 0:00:46.560 *********** 2020-07-13T06:07:16.366 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.031764", "end": "2020-07-13 06:07:16.551452", "rc": 0, "start": "2020-07-13 06:07:16.519688", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:07:16.428 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:16 +0000 (0:00:00.082) 0:00:46.643 *********** 2020-07-13T06:07:19.266 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** ok: [smithi014.front.sepia.ceph.com] => {"cache_update_time": 1594620381, "cache_updated": false, "changed": false} 2020-07-13T06:07:19.317 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:19 +0000 (0:00:02.889) 0:00:49.532 *********** 2020-07-13T06:07:19.326 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1594620377, "cache_updated": false, "changed": false} 2020-07-13T06:07:19.368 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:19 +0000 (0:00:00.052) 0:00:49.585 *********** 2020-07-13T06:07:28.515 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** changed: [smithi014.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 0 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 2020-07-13T06:07:28.518 INFO:teuthology.task.ansible.out: 95%\r(Reading database ... 100%\r(Reading database ... 102052 files and directories currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 0 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading d 2020-07-13T06:07:28.520 INFO:teuthology.task.ansible.out:atabase ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102052 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-07-13T06:07:28.561 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:28 +0000 (0:00:09.192) 0:00:58.777 *********** 2020-07-13T06:07:28.799 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 0 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102052 files and directories 2020-07-13T06:07:28.802 INFO:teuthology.task.ansible.out: currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 0 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-07-13T06:07:28.804 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102052 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-07-13T06:07:28.859 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:28 +0000 (0:00:00.297) 0:00:59.075 *********** 2020-07-13T06:07:29.497 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-07-13T06:07:29.499 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"cache_update_time": 1594620381, "cache_updated": false, "changed": false} 2020-07-13T06:07:29.552 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:29 +0000 (0:00:00.693) 0:00:59.768 *********** 2020-07-13T06:07:29.583 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-07-13T06:07:29.585 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:29.627 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:29 +0000 (0:00:00.074) 0:00:59.843 *********** 2020-07-13T06:07:29.769 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1594620377, "cache_updated": false, "changed": false} 2020-07-13T06:07:29.809 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:29 +0000 (0:00:00.182) 0:01:00.025 *********** 2020-07-13T06:07:29.853 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:29.888 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:29 +0000 (0:00:00.078) 0:01:00.104 *********** 2020-07-13T06:07:38.673 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-07-13T06:07:38.675 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => {"cache_update_time": 1594620381, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.2 [2723 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.2 [923 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.28 [39.6 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.28 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.28 [5188 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.28 [870 kB]\nFetched 10.4 MB in 1s (9670 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 2020-07-13T06:07:38.676 INFO:teuthology.task.ansible.out: 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101910 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.2_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.2_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu 2020-07-13T06:07:38.678 INFO:teuthology.task.ansible.out:1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.2 [2723 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.2 [923 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.28 [39.6 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.28 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.28 [5188 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.28 [870 kB]", "Fetched 10.4 MB in 1s (9670 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-07-13T06:07:38.680 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101910 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.2_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.2_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-07-13T06:07:38.716 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:38 +0000 (0:00:08.828) 0:01:08.932 *********** 2020-07-13T06:07:38.742 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1594620377, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.2 [2723 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.2 [923 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.28 [39.6 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.28 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.28 [5188 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.28 [870 kB]\nFetched 10.4 MB in 1s (9641 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 2020-07-13T06:07:38.744 INFO:teuthology.task.ansible.out: 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101910 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.2_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.2_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu 2020-07-13T06:07:38.746 INFO:teuthology.task.ansible.out:1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.2 [2723 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.2 [923 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.28 [39.6 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.28 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.28 [5188 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.28 [870 kB]", "Fetched 10.4 MB in 1s (9641 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-07-13T06:07:38.748 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101910 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.2_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.2_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-07-13T06:07:38.784 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:38 +0000 (0:00:00.068) 0:01:09.001 *********** 2020-07-13T06:07:39.638 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* ok: [smithi014.front.sepia.ceph.com] => {"cache_update_time": 1594620381, "cache_updated": false, "changed": false} 2020-07-13T06:07:39.682 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:39 +0000 (0:00:00.897) 0:01:09.898 *********** 2020-07-13T06:07:39.710 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1594620377, "cache_updated": false, "changed": false} 2020-07-13T06:07:39.747 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:39 +0000 (0:00:00.062) 0:01:09.960 *********** 2020-07-13T06:07:39.849 INFO:teuthology.task.ansible.out: 2020-07-13T06:07:39.851 INFO:teuthology.task.ansible.out:TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-07-13T06:07:39.896 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:39 +0000 (0:00:00.152) 0:01:10.113 *********** 2020-07-13T06:07:39.919 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-07-13T06:07:39.969 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:39 +0000 (0:00:00.072) 0:01:10.185 *********** 2020-07-13T06:07:39.980 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-07-13T06:07:39.981 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:40.024 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:40 +0000 (0:00:00.055) 0:01:10.240 *********** 2020-07-13T06:07:40.034 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:40.073 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:40 +0000 (0:00:00.048) 0:01:10.289 *********** 2020-07-13T06:07:40.192 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-07-13T06:07:40.194 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-07-13T06:07:40.242 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:40 +0000 (0:00:00.168) 0:01:10.458 *********** 2020-07-13T06:07:40.245 INFO:teuthology.task.ansible.out: 2020-07-13T06:07:40.274 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-07-13T06:07:40.312 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:40 +0000 (0:00:00.070) 0:01:10.529 *********** 2020-07-13T06:07:40.649 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-07-13T06:07:40.689 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:40 +0000 (0:00:00.376) 0:01:10.905 *********** 2020-07-13T06:07:40.724 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-07-13T06:07:40.766 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:40 +0000 (0:00:00.076) 0:01:10.982 *********** 2020-07-13T06:07:40.885 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ ok: [smithi014.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1222, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-07-13T06:07:40.932 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:40 +0000 (0:00:00.166) 0:01:11.148 *********** 2020-07-13T06:07:40.964 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1222, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-07-13T06:07:41.002 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:41 +0000 (0:00:00.070) 0:01:11.219 *********** 2020-07-13T06:07:41.085 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** ok: [smithi014.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-13T06:07:41.130 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:41 +0000 (0:00:00.127) 0:01:11.346 *********** 2020-07-13T06:07:41.161 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-13T06:07:41.199 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:41 +0000 (0:00:00.069) 0:01:11.415 *********** 2020-07-13T06:07:41.439 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* changed: [smithi014.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Mon 2020-07-13 06:05:37 UTC", "ActiveEnterTimestampMonotonic": "18625368", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target network.target remote-fs.target basic.target sysinit.target system.slice systemd-tmpfiles-setup.service systemd-journald.socket -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-07-13 06:05:36 UTC", "AssertTimestampMonotonic": "18211602", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-07-13 06:05:36 UTC", "ConditionTimestampMonotonic": "18211601", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", 2020-07-13T06:07:41.446 INFO:teuthology.task.ansible.out:"Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", "DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1146", "ExecMainStartTimestamp": "Mon 2020-07-13 06:05:37 UTC", "ExecMainStartTimestampMonotonic": "18625357", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-07-13 06:05:36 UTC", "InactiveExitTimestampMonotonic": "18212847", "InvocationID": "d35eb08275c54688bf826b4ee3fd70ac", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA 2020-07-13T06:07:41.457 INFO:teuthology.task.ansible.out:": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1146", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", 2020-07-13T06:07:41.459 INFO:teuthology.task.ansible.out: "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-07-13 06:05:37 UTC", "StateChangeTimestampMonotonic": "18625368", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-07-13 06:05:37 UTC", "WatchdogTimestampMonotonic": "18625367", "WatchdogUSec": "0"}} 2020-07-13T06:07:41.478 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:41 +0000 (0:00:00.278) 0:01:11.694 *********** 2020-07-13T06:07:41.532 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-07-13T06:07:41.535 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:41.575 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:41 +0000 (0:00:00.096) 0:01:11.791 *********** 2020-07-13T06:07:41.585 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-07-13T06:07:41.587 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Mon 2020-07-13 06:05:39 UTC", "ActiveEnterTimestampMonotonic": "18037331", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target remote-fs.target system.slice nss-lookup.target network.target -.mount systemd-journald.socket systemd-tmpfiles-setup.service sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-07-13 06:05:39 UTC", "AssertTimestampMonotonic": "17549321", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-07-13 06:05:39 UTC", "ConditionTimestampMonotonic": "17549321", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-07-13T06:07:41.589 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1209", "ExecMainStartTimestamp": "Mon 2020-07-13 06:05:39 UTC", "ExecMainStartTimestampMonotonic": "18037311", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-07-13 06:05:39 UTC", "InactiveExitTimestampMonotonic": "17550323", "InvocationID": "1256edb18b9e4c6181e07fddc5225e1e", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-07-13T06:07:41.590 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1209", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-07-13T06:07:41.592 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-07-13 06:05:39 UTC", "StateChangeTimestampMonotonic": "18037331", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-07-13 06:05:39 UTC", "WatchdogTimestampMonotonic": "18037329", "WatchdogUSec": "0"}} 2020-07-13T06:07:41.631 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:41 +0000 (0:00:00.054) 0:01:11.845 *********** 2020-07-13T06:07:41.639 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-07-13T06:07:41.643 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:41.682 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:41 +0000 (0:00:00.049) 0:01:11.895 *********** 2020-07-13T06:07:41.692 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:41.729 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:41 +0000 (0:00:00.050) 0:01:11.945 *********** 2020-07-13T06:07:41.738 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-07-13T06:07:41.739 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:41.780 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:41 +0000 (0:00:00.050) 0:01:11.996 *********** 2020-07-13T06:07:41.790 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:41.836 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:41 +0000 (0:00:00.052) 0:01:12.049 *********** 2020-07-13T06:07:41.850 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:41.886 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:41 +0000 (0:00:00.053) 0:01:12.102 *********** 2020-07-13T06:07:41.896 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-07-13T06:07:41.897 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:41.937 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:41 +0000 (0:00:00.051) 0:01:12.153 *********** 2020-07-13T06:07:41.951 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:41.987 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:41 +0000 (0:00:00.050) 0:01:12.203 *********** 2020-07-13T06:07:41.997 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-07-13T06:07:41.999 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.038 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.050) 0:01:12.254 *********** 2020-07-13T06:07:42.050 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-07-13T06:07:42.052 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.091 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.052) 0:01:12.307 *********** 2020-07-13T06:07:42.105 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.142 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.051) 0:01:12.358 *********** 2020-07-13T06:07:42.155 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-07-13T06:07:42.157 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.194 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.052) 0:01:12.411 *********** 2020-07-13T06:07:42.215 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.260 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.065) 0:01:12.476 *********** 2020-07-13T06:07:42.277 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.340 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.056) 0:01:12.533 *********** TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.375 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.058) 0:01:12.591 *********** 2020-07-13T06:07:42.385 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-13T06:07:42.387 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.428 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.052) 0:01:12.644 *********** 2020-07-13T06:07:42.437 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-07-13T06:07:42.440 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.483 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.050) 0:01:12.694 *********** 2020-07-13T06:07:42.488 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-07-13T06:07:42.489 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.527 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.049) 0:01:12.743 *********** 2020-07-13T06:07:42.537 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-13T06:07:42.539 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.577 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.050) 0:01:12.794 *********** 2020-07-13T06:07:42.589 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-07-13T06:07:42.591 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.634 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.053) 0:01:12.847 *********** 2020-07-13T06:07:42.640 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-07-13T06:07:42.647 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.683 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.050) 0:01:12.897 *********** 2020-07-13T06:07:42.694 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.738 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.056) 0:01:12.954 *********** 2020-07-13T06:07:42.749 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-07-13T06:07:42.751 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.797 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.053) 0:01:13.008 *********** 2020-07-13T06:07:42.805 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.841 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.049) 0:01:13.057 *********** 2020-07-13T06:07:42.872 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.895 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.053) 0:01:13.111 *********** 2020-07-13T06:07:42.903 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.941 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.046) 0:01:13.158 *********** 2020-07-13T06:07:42.951 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-07-13T06:07:42.953 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:42.989 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:42 +0000 (0:00:00.048) 0:01:13.206 *********** 2020-07-13T06:07:43.002 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.044 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.054) 0:01:13.260 *********** 2020-07-13T06:07:43.058 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.097 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.053) 0:01:13.314 *********** 2020-07-13T06:07:43.107 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-07-13T06:07:43.110 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.147 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.049) 0:01:13.363 *********** 2020-07-13T06:07:43.162 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.204 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.051) 0:01:13.415 *********** 2020-07-13T06:07:43.209 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.258 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.053) 0:01:13.468 *********** 2020-07-13T06:07:43.270 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.314 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.061) 0:01:13.530 *********** 2020-07-13T06:07:43.324 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-07-13T06:07:43.326 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.363 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.048) 0:01:13.578 *********** 2020-07-13T06:07:43.372 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-07-13T06:07:43.375 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.417 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.050) 0:01:13.629 *********** 2020-07-13T06:07:43.424 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.461 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.047) 0:01:13.677 *********** 2020-07-13T06:07:43.480 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.514 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.053) 0:01:13.730 *********** 2020-07-13T06:07:43.523 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-07-13T06:07:43.525 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.562 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.048) 0:01:13.779 *********** 2020-07-13T06:07:43.572 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-07-13T06:07:43.574 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.610 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.047) 0:01:13.826 *********** 2020-07-13T06:07:43.621 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-07-13T06:07:43.623 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.662 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.049) 0:01:13.876 *********** 2020-07-13T06:07:43.676 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-07-13T06:07:43.679 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.717 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.055) 0:01:13.931 *********** 2020-07-13T06:07:43.724 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-07-13T06:07:43.728 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.770 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.052) 0:01:13.984 *********** 2020-07-13T06:07:43.779 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-07-13T06:07:43.781 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.820 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.051) 0:01:14.036 *********** 2020-07-13T06:07:43.834 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.880 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.059) 0:01:14.096 *********** 2020-07-13T06:07:43.893 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-07-13T06:07:43.894 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.935 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.055) 0:01:14.151 *********** 2020-07-13T06:07:43.945 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-07-13T06:07:43.947 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:43.985 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:43 +0000 (0:00:00.049) 0:01:14.201 *********** 2020-07-13T06:07:43.993 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-07-13T06:07:43.997 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:44.032 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:44 +0000 (0:00:00.046) 0:01:14.248 *********** 2020-07-13T06:07:44.043 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-07-13T06:07:44.045 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:44.089 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:44 +0000 (0:00:00.056) 0:01:14.304 *********** 2020-07-13T06:07:44.100 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-07-13T06:07:44.102 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:44.150 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:44 +0000 (0:00:00.055) 0:01:14.359 *********** 2020-07-13T06:07:44.155 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:44.198 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:44 +0000 (0:00:00.052) 0:01:14.412 *********** 2020-07-13T06:07:44.209 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:44.252 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:44 +0000 (0:00:00.056) 0:01:14.469 *********** 2020-07-13T06:07:44.298 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:44.339 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:44 +0000 (0:00:00.087) 0:01:14.556 *********** 2020-07-13T06:07:44.349 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-07-13T06:07:44.351 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-07-13T06:07:44.389 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:44 +0000 (0:00:00.049) 0:01:14.605 *********** 2020-07-13T06:07:44.497 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-07-13T06:07:44.556 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:44 +0000 (0:00:00.166) 0:01:14.772 *********** 2020-07-13T06:07:44.794 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-07-13T06:07:44.840 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:44 +0000 (0:00:00.284) 0:01:15.056 *********** 2020-07-13T06:07:44.881 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-07-13T06:07:44.883 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:44.920 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:44 +0000 (0:00:00.079) 0:01:15.136 *********** 2020-07-13T06:07:44.973 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-07-13T06:07:44.976 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-07-13T06:07:45.014 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:45 +0000 (0:00:00.094) 0:01:15.230 *********** 2020-07-13T06:07:45.065 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-07-13T06:07:45.067 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:45.108 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:45 +0000 (0:00:00.092) 0:01:15.323 *********** 2020-07-13T06:07:45.329 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-07-13T06:07:45.367 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:45 +0000 (0:00:00.260) 0:01:15.584 *********** 2020-07-13T06:07:45.586 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-07-13T06:07:45.636 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:45 +0000 (0:00:00.268) 0:01:15.852 *********** 2020-07-13T06:07:45.739 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-07-13T06:07:45.741 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-07-13T06:07:45.778 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:45 +0000 (0:00:00.142) 0:01:15.994 *********** 2020-07-13T06:07:46.044 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-07-13T06:07:46.083 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:46 +0000 (0:00:00.304) 0:01:16.299 *********** 2020-07-13T06:07:46.186 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-07-13T06:07:46.188 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-07-13T06:07:46.236 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:46 +0000 (0:00:00.152) 0:01:16.451 *********** 2020-07-13T06:07:46.394 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-07-13T06:07:46.396 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-13T06:07:46.435 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:46 +0000 (0:00:00.199) 0:01:16.651 *********** 2020-07-13T06:07:46.463 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-07-13T06:07:46.508 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:46 +0000 (0:00:00.069) 0:01:16.720 *********** 2020-07-13T06:07:46.514 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-07-13T06:07:46.516 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:46.568 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:46 +0000 (0:00:00.062) 0:01:16.783 *********** 2020-07-13T06:07:46.600 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:46.638 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:46 +0000 (0:00:00.070) 0:01:16.853 *********** 2020-07-13T06:07:46.661 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-07-13T06:07:46.663 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-13T06:07:46.699 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:46 +0000 (0:00:00.061) 0:01:16.915 *********** 2020-07-13T06:07:46.709 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-07-13T06:07:46.711 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:46.760 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:46 +0000 (0:00:00.061) 0:01:16.977 *********** 2020-07-13T06:07:46.771 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-07-13T06:07:46.773 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:46.812 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:46 +0000 (0:00:00.051) 0:01:17.028 *********** 2020-07-13T06:07:46.826 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:46.863 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:46 +0000 (0:00:00.051) 0:01:17.079 *********** 2020-07-13T06:07:46.875 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-07-13T06:07:46.877 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:46.918 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:46 +0000 (0:00:00.054) 0:01:17.134 *********** 2020-07-13T06:07:46.928 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-07-13T06:07:46.931 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:46.970 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:46 +0000 (0:00:00.051) 0:01:17.186 *********** 2020-07-13T06:07:46.981 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-07-13T06:07:46.983 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.020 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:47 +0000 (0:00:00.050) 0:01:17.236 *********** 2020-07-13T06:07:47.033 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-13T06:07:47.036 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-07-13T06:07:47.072 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:47 +0000 (0:00:00.052) 0:01:17.289 *********** 2020-07-13T06:07:47.084 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-07-13T06:07:47.086 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.124 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:47 +0000 (0:00:00.051) 0:01:17.340 *********** 2020-07-13T06:07:47.148 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi014.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.162 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.164 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.206 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:47 +0000 (0:00:00.082) 0:01:17.423 *********** 2020-07-13T06:07:47.217 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-07-13T06:07:47.218 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} ok: [smithi014.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-07-13T06:07:47.220 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.221 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.222 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.232 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.234 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.255 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi014.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.265 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.301 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:47 +0000 (0:00:00.094) 0:01:17.517 *********** 2020-07-13T06:07:47.350 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:47 +0000 (0:00:00.047) 0:01:17.565 *********** 2020-07-13T06:07:47.421 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.432 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.434 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.444 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.446 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.458 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.468 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-07-13T06:07:47.479 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.490 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.492 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.506 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.514 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.525 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.538 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.603 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:47 +0000 (0:00:00.253) 0:01:17.819 *********** 2020-07-13T06:07:47.613 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-07-13T06:07:47.615 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004316", "end": "2020-07-13 06:07:47.552498", "rc": 0, "start": "2020-07-13 06:07:47.548182", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-07-13T06:07:47.657 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:47 +0000 (0:00:00.053) 0:01:17.873 *********** 2020-07-13T06:07:47.698 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-07-13T06:07:47.700 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.740 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:47 +0000 (0:00:00.083) 0:01:17.956 *********** 2020-07-13T06:07:47.794 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.838 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:47 +0000 (0:00:00.094) 0:01:18.051 *********** 2020-07-13T06:07:47.868 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-07-13T06:07:47.870 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004598", "end": "2020-07-13 06:07:47.839148", "rc": 0, "start": "2020-07-13 06:07:47.834550", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-07-13T06:07:47.918 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:47 +0000 (0:00:00.082) 0:01:18.133 *********** 2020-07-13T06:07:47.936 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-07-13T06:07:47.941 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:47.987 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:47 +0000 (0:00:00.064) 0:01:18.197 *********** 2020-07-13T06:07:47.998 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:48.051 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:48 +0000 (0:00:00.066) 0:01:18.264 *********** 2020-07-13T06:07:48.088 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-07-13T06:07:48.090 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:48.158 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:48 +0000 (0:00:00.109) 0:01:18.374 *********** 2020-07-13T06:07:48.194 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:48.233 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:48 +0000 (0:00:00.075) 0:01:18.449 *********** 2020-07-13T06:07:48.891 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-07-13T06:07:48.894 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"cache_update_time": 1594620381, "cache_updated": false, "changed": false} 2020-07-13T06:07:48.932 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:48 +0000 (0:00:00.698) 0:01:19.148 *********** 2020-07-13T06:07:48.987 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:49.054 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:49 +0000 (0:00:00.121) 0:01:19.270 *********** 2020-07-13T06:07:49.109 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-07-13T06:07:49.111 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => (item={'block_used': 13634344, 'uuid': '21f1a39c-c8bb-409b-ab72-745e0ce7d061', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226460764, 'size_available': 927583289344, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113835, 'block_size': 4096, 'inode_available': 60941141}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226460764, "block_size": 4096, "block_total": 240095108, "block_used": 13634344, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941141, "inode_total": 61054976, "inode_used": 113835, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927583289344, "size_total": 983429562368, "uuid": "21f1a39c-c8bb-409b-ab72-745e0ce7d061"}} 2020-07-13T06:07:49.157 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:49 +0000 (0:00:00.099) 0:01:19.370 *********** 2020-07-13T06:07:49.176 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-07-13T06:07:49.178 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1594620377, "cache_updated": false, "changed": false} 2020-07-13T06:07:49.234 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:49 +0000 (0:00:00.076) 0:01:19.446 *********** 2020-07-13T06:07:49.265 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:49.314 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:49 +0000 (0:00:00.077) 0:01:19.523 *********** 2020-07-13T06:07:49.322 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi014.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.011048", "end": "2020-07-13 06:07:49.509674", "rc": 0, "start": "2020-07-13 06:07:49.498626", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:07:49.363 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:49 +0000 (0:00:00.054) 0:01:19.578 *********** 2020-07-13T06:07:49.378 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi017.front.sepia.ceph.com] => (item={'block_used': 13634343, 'uuid': '21f1a39c-c8bb-409b-ab72-745e0ce7d061', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226460765, 'size_available': 927583293440, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113835, 'block_size': 4096, 'inode_available': 60941141}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226460765, "block_size": 4096, "block_total": 240095108, "block_used": 13634343, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941141, "inode_total": 61054976, "inode_used": 113835, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927583293440, "size_total": 983429562368, "uuid": "21f1a39c-c8bb-409b-ab72-745e0ce7d061"}} 2020-07-13T06:07:49.424 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:49 +0000 (0:00:00.061) 0:01:19.640 *********** 2020-07-13T06:07:49.434 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-07-13T06:07:49.436 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'block_used': 13634344, 'uuid': '21f1a39c-c8bb-409b-ab72-745e0ce7d061', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226460764, 'size_available': 927583289344, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113835, 'block_size': 4096, 'inode_available': 60941141}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226460764, "block_size": 4096, "block_total": 240095108, "block_used": 13634344, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941141, "inode_total": 61054976, "inode_used": 113835, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927583289344, "size_total": 983429562368, "uuid": "21f1a39c-c8bb-409b-ab72-745e0ce7d061"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:49.486 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:49 +0000 (0:00:00.062) 0:01:19.703 *********** 2020-07-13T06:07:49.523 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-07-13T06:07:49.563 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:49 +0000 (0:00:00.077) 0:01:19.780 *********** 2020-07-13T06:07:49.586 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-07-13T06:07:49.588 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009819", "end": "2020-07-13 06:07:48.945278", "rc": 0, "start": "2020-07-13 06:07:48.935459", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:07:49.654 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:49 +0000 (0:00:00.089) 0:01:19.870 *********** 2020-07-13T06:07:49.670 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-07-13T06:07:49.673 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:49.728 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi017.front.sepia.ceph.com] => (item={'block_used': 13634343, 'uuid': '21f1a39c-c8bb-409b-ab72-745e0ce7d061', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226460765, 'size_available': 927583293440, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113835, 'block_size': 4096, 'inode_available': 60941141}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226460765, "block_size": 4096, "block_total": 240095108, "block_used": 13634343, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941141, "inode_total": 61054976, "inode_used": 113835, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927583293440, "size_total": 983429562368, "uuid": "21f1a39c-c8bb-409b-ab72-745e0ce7d061"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:49.772 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:49 +0000 (0:00:00.118) 0:01:19.989 *********** 2020-07-13T06:07:49.834 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-07-13T06:07:49.882 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:49 +0000 (0:00:00.105) 0:01:20.094 *********** 2020-07-13T06:07:49.940 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-07-13T06:07:49.942 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:51.387 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4Lwo9XUuL9VYXBPZG4cU5H5eNceuag49CX3'], 'uuids': ['3edc3735-453a-4ea8-b649-cbb529384c73']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.604293", "end": "2020-07-13 06:07:51.572537", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4Lwo9XUuL9VYXBPZG4cU5H5eNceuag49CX3"], "labels": [], "masters": [], "uuids": ["3edc3735-453a-4ea8-b649-cbb529384c73"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:49.968244", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-13T06:07:51.398 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:51.399 INFO:teuthology.task.ansible.out: 2020-07-13T06:07:51.411 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:51.965 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmIIWOn4rtl3HnayGkIilnt1UQtfxAPde8rx'], 'uuids': ['980d4652-cd59-42bc-aa46-ae713a8b58f5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.885998", "end": "2020-07-13 06:07:51.312983", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmIIWOn4rtl3HnayGkIilnt1UQtfxAPde8rx"], "labels": [], "masters": [], "uuids": ["980d4652-cd59-42bc-aa46-ae713a8b58f5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:49.426985", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-13T06:07:51.978 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:51.989 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:52.687 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4LwrXA0n1EFnZBPt7eWY2TDibwWX1crkysV'], 'uuids': ['32d48f69-ed60-4f62-9c15-a1e5f728d8c8']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.122302", "end": "2020-07-13 06:07:52.868438", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4LwrXA0n1EFnZBPt7eWY2TDibwWX1crkysV"], "labels": [], "masters": [], "uuids": ["32d48f69-ed60-4f62-9c15-a1e5f728d8c8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:51.746136", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-13T06:07:52.700 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPL1', 'wwn-0x5000c500871ab133'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '21f1a39c-c8bb-409b-ab72-745e0ce7d061', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPL1-part1', 'wwn-0x5000c500871ab133-part1'], 'uuids': ['21f1a39c-c8bb-409b-ab72-745e0ce7d061']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871ab133', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HPL1', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPL1", "wwn-0x5000c500871ab133"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPL1-part1", "wwn-0x5000c500871ab133-part1"], "labels": [], "masters": [], "uuids": ["21f1a39c-c8bb-409b-ab72-745e0ce7d061"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "21f1a39c-c8bb-409b-ab72-745e0ce7d061"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HPL1", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871ab133"}}, "skip_reason": 2020-07-13T06:07:52.722 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-07-13T06:07:53.254 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmIIJXaVVW0gsgaLNk8uQKnSy4ugbDWfl2Jk'], 'uuids': ['22184078-ab38-49bd-be24-a5e1beb2580b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.127830", "end": "2020-07-13 06:07:52.594107", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmIIJXaVVW0gsgaLNk8uQKnSy4ugbDWfl2Jk"], "labels": [], "masters": [], "uuids": ["22184078-ab38-49bd-be24-a5e1beb2580b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:51.466277", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-13T06:07:53.255 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2SL', 'wwn-0x5000c500871b6e94'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '21f1a39c-c8bb-409b-ab72-745e0ce7d061', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2SL-part1', 'wwn-0x5000c500871b6e94-part1'], 'uuids': ['21f1a39c-c8bb-409b-ab72-745e0ce7d061']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b6e94', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4J2SL', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2SL", "wwn-0x5000c500871b6e94"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2SL-part1", "wwn-0x5000c500871b6e94-part1"], "labels": [], "masters": [], "uuids": ["21f1a39c-c8bb-409b-ab72-745e0ce7d061"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "21f1a39c-c8bb-409b-ab72-745e0ce7d061"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J2SL", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b6e94"}}, "skip_reason": 2020-07-13T06:07:53.257 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-07-13T06:07:53.985 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-45jlec-Vncy-ZGhm-I9ik-XIEl-pSe8-NmxsVW', 'nvme-INTEL_SSDPEDMD400G4_CVFT53300003400BGN', 'nvme-nvme.8086-43564654353333303030303334303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.139598", "end": "2020-07-13 06:07:54.165243", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-45jlec-Vncy-ZGhm-I9ik-XIEl-pSe8-NmxsVW", "nvme-INTEL_SSDPEDMD400G4_CVFT53300003400BGN", "nvme-nvme.8086-43564654353333303030303334303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:53.025645", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-07-13T06:07:53.989 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-13T06:07:54.000 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:54.014 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:54.585 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-QcsAXZ-6nYI-Ye04-3VI3-6R3H-eeMf-wbyvHB', 'nvme-INTEL_SSDPEDMD400G4_CVFT533000D8400BGN', 'nvme-nvme.8086-43564654353333303030443834303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.152386", "end": "2020-07-13 06:07:53.909265", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-QcsAXZ-6nYI-Ye04-3VI3-6R3H-eeMf-wbyvHB", "nvme-INTEL_SSDPEDMD400G4_CVFT533000D8400BGN", "nvme-nvme.8086-43564654353333303030443834303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:52.756879", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-07-13T06:07:54.589 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-13T06:07:54.618 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:54.631 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:55.299 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4Lw0lYCZNzaFhuPepqM9n30VAcWJfKv5PDd'], 'uuids': ['53e8ac35-9299-4bfd-9dab-a1b2040eb9f1']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.134113", "end": "2020-07-13 06:07:55.484785", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4Lw0lYCZNzaFhuPepqM9n30VAcWJfKv5PDd"], "labels": [], "masters": [], "uuids": ["53e8ac35-9299-4bfd-9dab-a1b2040eb9f1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:54.350672", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-13T06:07:55.326 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:55.340 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:55.898 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmIIW20yfKowC9Q0dZ0GIYfD9qnO9mh5Or3L'], 'uuids': ['d5192ae3-6d60-4206-9a5e-3df418695f33']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.141302", "end": "2020-07-13 06:07:55.255194", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmIIW20yfKowC9Q0dZ0GIYfD9qnO9mh5Or3L"], "labels": [], "masters": [], "uuids": ["d5192ae3-6d60-4206-9a5e-3df418695f33"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:54.113892", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-13T06:07:55.909 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:55.923 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:56.582 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4LwrrDPvbDGOnbHWnuJfFPV33vtwaG5wNZu'], 'uuids': ['21784476-8f30-4627-940c-314dd48081cf']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.114982", "end": "2020-07-13 06:07:56.768494", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4LwrrDPvbDGOnbHWnuJfFPV33vtwaG5wNZu"], "labels": [], "masters": [], "uuids": ["21784476-8f30-4627-940c-314dd48081cf"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:55.653512", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-13T06:07:56.609 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:57.173 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmII0tBvSstDu88WCUkZZkfBsCsPDf1bCtBS'], 'uuids': ['159d01c7-940e-4a90-97dc-1635e2926cfa']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.120541", "end": "2020-07-13 06:07:56.529164", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmII0tBvSstDu88WCUkZZkfBsCsPDf1bCtBS"], "labels": [], "masters": [], "uuids": ["159d01c7-940e-4a90-97dc-1635e2926cfa"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:55.408623", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-13T06:07:57.184 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:57.861 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4LwjQm1yNF7ict3ruqlti4ca0LC4VgTyXDb'], 'uuids': ['ef65654a-4007-4f3e-afbe-5d9a3bba1699']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.106402", "end": "2020-07-13 06:07:58.044397", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4LwjQm1yNF7ict3ruqlti4ca0LC4VgTyXDb"], "labels": [], "masters": [], "uuids": ["ef65654a-4007-4f3e-afbe-5d9a3bba1699"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:56.937995", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-13T06:07:57.932 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:57 +0000 (0:00:08.054) 0:01:28.149 *********** 2020-07-13T06:07:58.009 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-07-13T06:07:58.011 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:58.289 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4Lwo9XUuL9VYXBPZG4cU5H5eNceuag49CX3'], 'uuids': ['3edc3735-453a-4ea8-b649-cbb529384c73']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.153836", "end": "2020-07-13 06:07:58.475341", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4Lwo9XUuL9VYXBPZG4cU5H5eNceuag49CX3"], "labels": [], "masters": [], "uuids": ["3edc3735-453a-4ea8-b649-cbb529384c73"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:58.321505", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142805 s, 808 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142805 s, 808 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-13T06:07:58.301 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:58.315 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:58.460 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-07-13T06:07:58.462 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmIItIWAlMZRIJPt6Ny8P7WQvQEmKMRFi2bM'], 'uuids': ['9a96efb1-36ed-4962-82db-dd0b012be17a']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.110644", "end": "2020-07-13 06:07:57.804432", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmIItIWAlMZRIJPt6Ny8P7WQvQEmKMRFi2bM"], "labels": [], "masters": [], "uuids": ["9a96efb1-36ed-4962-82db-dd0b012be17a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:56.693788", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-13T06:07:58.521 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:58 +0000 (0:00:00.588) 0:01:28.737 *********** 2020-07-13T06:07:58.576 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-07-13T06:07:58.578 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:58.611 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4LwrXA0n1EFnZBPt7eWY2TDibwWX1crkysV'], 'uuids': ['32d48f69-ed60-4f62-9c15-a1e5f728d8c8']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.154321", "end": "2020-07-13 06:07:58.774682", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4LwrXA0n1EFnZBPt7eWY2TDibwWX1crkysV"], "labels": [], "masters": [], "uuids": ["32d48f69-ed60-4f62-9c15-a1e5f728d8c8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:58.620361", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149727 s, 770 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149727 s, 770 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-13T06:07:58.613 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPL1', 'wwn-0x5000c500871ab133'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '21f1a39c-c8bb-409b-ab72-745e0ce7d061', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPL1-part1', 'wwn-0x5000c500871ab133-part1'], 'uuids': ['21f1a39c-c8bb-409b-ab72-745e0ce7d061']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871ab133', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HPL1', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPL1", "wwn-0x5000c500871ab133"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPL1-part1", "wwn-0x5000c500871ab133-part1"], "labels": [], "masters": [], "uuids": ["21f1a39c-c8bb-409b-ab72-745e0ce7d061"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "21f1a39c-c8bb-409b-ab72-745e0ce7d061"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HPL1", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871ab133"}}, "skip_reason": 2020-07-13T06:07:58.615 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-07-13T06:07:58.811 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-45jlec-Vncy-ZGhm-I9ik-XIEl-pSe8-NmxsVW', 'nvme-INTEL_SSDPEDMD400G4_CVFT53300003400BGN', 'nvme-nvme.8086-43564654353333303030303334303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.052858", "end": "2020-07-13 06:07:58.995042", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-45jlec-Vncy-ZGhm-I9ik-XIEl-pSe8-NmxsVW", "nvme-INTEL_SSDPEDMD400G4_CVFT53300003400BGN", "nvme-nvme.8086-43564654353333303030303334303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:58.942184", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.048086 s, 2.4 GB/s", "stde 2020-07-13T06:07:58.813 INFO:teuthology.task.ansible.out:rr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.048086 s, 2.4 GB/s"], "stdout": "", "stdout_lines": []} 2020-07-13T06:07:58.822 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:58.835 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:58.882 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmIIWOn4rtl3HnayGkIilnt1UQtfxAPde8rx'], 'uuids': ['980d4652-cd59-42bc-aa46-ae713a8b58f5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.151863", "end": "2020-07-13 06:07:58.222038", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmIIWOn4rtl3HnayGkIilnt1UQtfxAPde8rx"], "labels": [], "masters": [], "uuids": ["980d4652-cd59-42bc-aa46-ae713a8b58f5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:58.070175", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138535 s, 833 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138535 s, 833 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-13T06:07:58.893 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:58.905 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:59.121 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4Lw0lYCZNzaFhuPepqM9n30VAcWJfKv5PDd'], 'uuids': ['53e8ac35-9299-4bfd-9dab-a1b2040eb9f1']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.149485", "end": "2020-07-13 06:07:59.301254", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4Lw0lYCZNzaFhuPepqM9n30VAcWJfKv5PDd"], "labels": [], "masters": [], "uuids": ["53e8ac35-9299-4bfd-9dab-a1b2040eb9f1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:59.151769", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.146496 s, 787 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.146496 s, 787 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-13T06:07:59.132 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:59.150 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:59.199 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmIIJXaVVW0gsgaLNk8uQKnSy4ugbDWfl2Jk'], 'uuids': ['22184078-ab38-49bd-be24-a5e1beb2580b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.148709", "end": "2020-07-13 06:07:58.545141", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmIIJXaVVW0gsgaLNk8uQKnSy4ugbDWfl2Jk"], "labels": [], "masters": [], "uuids": ["22184078-ab38-49bd-be24-a5e1beb2580b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:58.396432", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144967 s, 796 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144967 s, 796 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-13T06:07:59.222 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2SL', 'wwn-0x5000c500871b6e94'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '21f1a39c-c8bb-409b-ab72-745e0ce7d061', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2SL-part1', 'wwn-0x5000c500871b6e94-part1'], 'uuids': ['21f1a39c-c8bb-409b-ab72-745e0ce7d061']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b6e94', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4J2SL', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2SL", "wwn-0x5000c500871b6e94"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2SL-part1", "wwn-0x5000c500871b6e94-part1"], "labels": [], "masters": [], "uuids": ["21f1a39c-c8bb-409b-ab72-745e0ce7d061"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "21f1a39c-c8bb-409b-ab72-745e0ce7d061"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J2SL", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b6e94"}}, "skip_reason": 2020-07-13T06:07:59.224 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-07-13T06:07:59.378 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-QcsAXZ-6nYI-Ye04-3VI3-6R3H-eeMf-wbyvHB', 'nvme-INTEL_SSDPEDMD400G4_CVFT533000D8400BGN', 'nvme-nvme.8086-43564654353333303030443834303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.053307", "end": "2020-07-13 06:07:58.738135", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-QcsAXZ-6nYI-Ye04-3VI3-6R3H-eeMf-wbyvHB", "nvme-INTEL_SSDPEDMD400G4_CVFT533000D8400BGN", "nvme-nvme.8086-43564654353333303030443834303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:58.684828", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0485922 s, 2.4 GB/s", "std 2020-07-13T06:07:59.380 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0485922 s, 2.4 GB/s"], "stdout": "", "stdout_lines": []} 2020-07-13T06:07:59.401 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:59.412 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:59.445 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4LwrrDPvbDGOnbHWnuJfFPV33vtwaG5wNZu'], 'uuids': ['21784476-8f30-4627-940c-314dd48081cf']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.156580", "end": "2020-07-13 06:07:59.626989", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4LwrrDPvbDGOnbHWnuJfFPV33vtwaG5wNZu"], "labels": [], "masters": [], "uuids": ["21784476-8f30-4627-940c-314dd48081cf"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:59.470409", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151783 s, 760 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151783 s, 760 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-13T06:07:59.491 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:59.744 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmIIW20yfKowC9Q0dZ0GIYfD9qnO9mh5Or3L'], 'uuids': ['d5192ae3-6d60-4206-9a5e-3df418695f33']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.157322", "end": "2020-07-13 06:07:59.082359", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmIIW20yfKowC9Q0dZ0GIYfD9qnO9mh5Or3L"], "labels": [], "masters": [], "uuids": ["d5192ae3-6d60-4206-9a5e-3df418695f33"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:58.925037", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153469 s, 752 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153469 s, 752 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-13T06:07:59.766 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:59.778 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:07:59.802 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4LwjQm1yNF7ict3ruqlti4ca0LC4VgTyXDb'], 'uuids': ['ef65654a-4007-4f3e-afbe-5d9a3bba1699']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.151074", "end": "2020-07-13 06:07:59.972355", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-TxL2x9pRTuqdyhkwqfMiTSvu3upMN4LwjQm1yNF7ict3ruqlti4ca0LC4VgTyXDb"], "labels": [], "masters": [], "uuids": ["ef65654a-4007-4f3e-afbe-5d9a3bba1699"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:59.821281", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.146213 s, 789 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.146213 s, 789 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-13T06:07:59.877 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:07:59 +0000 (0:00:01.356) 0:01:30.093 *********** 2020-07-13T06:08:00.061 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmII0tBvSstDu88WCUkZZkfBsCsPDf1bCtBS'], 'uuids': ['159d01c7-940e-4a90-97dc-1635e2926cfa']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.150820", "end": "2020-07-13 06:07:59.410147", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmII0tBvSstDu88WCUkZZkfBsCsPDf1bCtBS"], "labels": [], "masters": [], "uuids": ["159d01c7-940e-4a90-97dc-1635e2926cfa"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:59.259327", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.147918 s, 780 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.147918 s, 780 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-13T06:08:00.073 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:00.174 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-07-13T06:08:00.180 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.147361", "end": "2020-07-13 06:08:00.361083", "rc": 0, "start": "2020-07-13 06:08:00.213722", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:08:00.233 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:00 +0000 (0:00:00.355) 0:01:30.449 *********** 2020-07-13T06:08:00.372 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-07-13T06:08:00.374 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmIItIWAlMZRIJPt6Ny8P7WQvQEmKMRFi2bM'], 'uuids': ['9a96efb1-36ed-4962-82db-dd0b012be17a']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.152663", "end": "2020-07-13 06:07:59.713816", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-sDpvqa0EdX92CRCUKTtErTRYbUs5hmIItIWAlMZRIJPt6Ny8P7WQvQEmKMRFi2bM"], "labels": [], "masters": [], "uuids": ["9a96efb1-36ed-4962-82db-dd0b012be17a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-13 06:07:59.561153", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.147876 s, 780 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.147876 s, 780 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-13T06:08:00.432 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:00 +0000 (0:00:00.199) 0:01:30.648 *********** 2020-07-13T06:08:00.455 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-07-13T06:08:00.456 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.060908", "end": "2020-07-13 06:08:00.653439", "rc": 0, "start": "2020-07-13 06:08:00.592531", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:08:00.514 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:00 +0000 (0:00:00.076) 0:01:30.724 *********** 2020-07-13T06:08:00.547 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-07-13T06:08:00.596 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:00 +0000 (0:00:00.088) 0:01:30.812 *********** 2020-07-13T06:08:00.650 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-07-13T06:08:00.652 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:00.692 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:00 +0000 (0:00:00.095) 0:01:30.908 *********** 2020-07-13T06:08:00.733 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-07-13T06:08:00.737 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:00.773 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:00 +0000 (0:00:00.078) 0:01:30.986 *********** 2020-07-13T06:08:00.780 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-07-13T06:08:00.782 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.164201", "end": "2020-07-13 06:08:00.101122", "rc": 0, "start": "2020-07-13 06:07:59.936921", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:08:00.830 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:00 +0000 (0:00:00.059) 0:01:31.046 *********** 2020-07-13T06:08:00.842 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:00.902 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:00 +0000 (0:00:00.063) 0:01:31.110 *********** 2020-07-13T06:08:00.936 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-07-13T06:08:00.938 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:00.979 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:00 +0000 (0:00:00.079) 0:01:31.190 *********** 2020-07-13T06:08:01.021 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-07-13T06:08:01.023 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.046315", "end": "2020-07-13 06:08:00.381438", "rc": 0, "start": "2020-07-13 06:08:00.335123", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:08:01.068 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:01 +0000 (0:00:00.093) 0:01:31.284 *********** 2020-07-13T06:08:01.117 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi014.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-13T06:08:01.158 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:01 +0000 (0:00:00.090) 0:01:31.374 *********** 2020-07-13T06:08:01.200 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:01 +0000 (0:00:00.042) 0:01:31.417 *********** 2020-07-13T06:08:01.213 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-07-13T06:08:01.214 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'block_used': 13634344, 'uuid': '21f1a39c-c8bb-409b-ab72-745e0ce7d061', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226460764, 'size_available': 927583289344, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113835, 'block_size': 4096, 'inode_available': 60941141}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226460764, "block_size": 4096, "block_total": 240095108, "block_used": 13634344, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941141, "inode_total": 61054976, "inode_used": 113835, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927583289344, "size_total": 983429562368, "uuid": "21f1a39c-c8bb-409b-ab72-745e0ce7d061"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:01.255 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:01 +0000 (0:00:00.054) 0:01:31.471 *********** 2020-07-13T06:08:01.266 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-07-13T06:08:01.268 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:01.322 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:01 +0000 (0:00:00.067) 0:01:31.539 *********** 2020-07-13T06:08:01.332 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-07-13T06:08:01.334 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:01.372 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:01 +0000 (0:00:00.049) 0:01:31.588 *********** 2020-07-13T06:08:01.385 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-07-13T06:08:01.387 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:01.423 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:01 +0000 (0:00:00.051) 0:01:31.640 *********** 2020-07-13T06:08:01.434 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-13T06:08:01.435 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:01.474 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:01 +0000 (0:00:00.050) 0:01:31.690 *********** 2020-07-13T06:08:01.483 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-07-13T06:08:01.485 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:01.536 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:01 +0000 (0:00:00.061) 0:01:31.752 *********** 2020-07-13T06:08:01.546 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-13T06:08:01.548 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:01.587 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:01 +0000 (0:00:00.050) 0:01:31.803 *********** 2020-07-13T06:08:01.598 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-07-13T06:08:01.600 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:01.644 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:01 +0000 (0:00:00.057) 0:01:31.860 *********** 2020-07-13T06:08:01.653 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-07-13T06:08:01.655 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:01.696 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:01 +0000 (0:00:00.051) 0:01:31.912 *********** 2020-07-13T06:08:01.749 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-07-13T06:08:01.751 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:01.803 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:01 +0000 (0:00:00.107) 0:01:32.019 *********** 2020-07-13T06:08:01.814 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-07-13T06:08:01.816 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-13T06:08:01.862 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:01 +0000 (0:00:00.059) 0:01:32.079 *********** 2020-07-13T06:08:01.873 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-07-13T06:08:01.874 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:01.917 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:01 +0000 (0:00:00.054) 0:01:32.133 *********** 2020-07-13T06:08:01.935 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-07-13T06:08:01.937 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'block_used': 13634343, 'uuid': '21f1a39c-c8bb-409b-ab72-745e0ce7d061', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226460765, 'size_available': 927583293440, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113835, 'block_size': 4096, 'inode_available': 60941141}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226460765, "block_size": 4096, "block_total": 240095108, "block_used": 13634343, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941141, "inode_total": 61054976, "inode_used": 113835, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927583293440, "size_total": 983429562368, "uuid": "21f1a39c-c8bb-409b-ab72-745e0ce7d061"}, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:01.981 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:01 +0000 (0:00:00.064) 0:01:32.197 *********** 2020-07-13T06:08:02.023 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:02.062 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:02 +0000 (0:00:00.081) 0:01:32.279 *********** 2020-07-13T06:08:02.117 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:02.179 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:02 +0000 (0:00:00.116) 0:01:32.395 *********** 2020-07-13T06:08:02.223 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-13T06:08:02.225 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:02.271 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:02 +0000 (0:00:00.091) 0:01:32.487 *********** 2020-07-13T06:08:02.312 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-07-13T06:08:02.314 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:02.359 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:02 +0000 (0:00:00.087) 0:01:32.575 *********** 2020-07-13T06:08:02.391 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-07-13T06:08:02.393 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:02.431 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:02 +0000 (0:00:00.072) 0:01:32.647 *********** 2020-07-13T06:08:02.462 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-07-13T06:08:02.465 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:02.515 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:02 +0000 (0:00:00.083) 0:01:32.731 *********** 2020-07-13T06:08:02.631 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-07-13T06:08:02.633 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-07-13T06:08:02.688 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:02 +0000 (0:00:00.172) 0:01:32.904 *********** 2020-07-13T06:08:02.880 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-07-13T06:08:02.921 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:02 +0000 (0:00:00.232) 0:01:33.137 *********** 2020-07-13T06:08:03.312 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-13T06:08:03.334 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-13T06:08:03.571 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-13T06:08:03.638 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-13T06:08:03.861 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-13T06:08:03.896 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-13T06:08:04.115 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-13T06:08:04.164 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-13T06:08:04.388 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-13T06:08:04.445 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:04 +0000 (0:00:01.524) 0:01:34.661 *********** 2020-07-13T06:08:04.457 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-13T06:08:04.503 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:04 +0000 (0:00:00.057) 0:01:34.719 *********** 2020-07-13T06:08:04.604 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-07-13T06:08:04.606 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-07-13T06:08:04.643 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:04 +0000 (0:00:00.140) 0:01:34.860 *********** 2020-07-13T06:08:04.665 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-07-13T06:08:04.704 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:04 +0000 (0:00:00.060) 0:01:34.920 *********** 2020-07-13T06:08:04.829 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-07-13T06:08:04.830 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-13T06:08:04.894 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-13T06:08:04.960 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-13T06:08:05.022 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-13T06:08:05.098 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-13T06:08:05.160 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-13T06:08:05.235 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-13T06:08:05.245 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:05.299 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:05 +0000 (0:00:00.595) 0:01:35.516 *********** 2020-07-13T06:08:05.310 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-13T06:08:05.311 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:05.364 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:05 +0000 (0:00:00.064) 0:01:35.580 *********** 2020-07-13T06:08:05.454 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-07-13T06:08:05.456 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-07-13T06:08:05.511 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:05 +0000 (0:00:00.146) 0:01:35.727 *********** 2020-07-13T06:08:05.525 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-07-13T06:08:05.567 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:05 +0000 (0:00:00.056) 0:01:35.783 *********** 2020-07-13T06:08:05.577 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-07-13T06:08:05.579 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-07-13T06:08:05.633 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:05 +0000 (0:00:00.065) 0:01:35.849 *********** 2020-07-13T06:08:05.650 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-07-13T06:08:05.688 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:05 +0000 (0:00:00.054) 0:01:35.904 *********** 2020-07-13T06:08:06.180 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-07-13T06:08:06.182 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => {"changed": true} 2020-07-13T06:08:06.238 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:06 +0000 (0:00:00.550) 0:01:36.454 *********** 2020-07-13T06:08:06.249 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true} 2020-07-13T06:08:06.298 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:06 +0000 (0:00:00.052) 0:01:36.507 *********** 2020-07-13T06:08:06.764 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-07-13T06:08:06.766 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-07-13T06:08:06.808 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:06 +0000 (0:00:00.517) 0:01:37.024 *********** 2020-07-13T06:08:06.858 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:06 +0000 (0:00:00.050) 0:01:37.074 *********** 2020-07-13T06:08:06.990 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-07-13T06:08:06.992 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "gid": 1222, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-07-13T06:08:07.060 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:07 +0000 (0:00:00.196) 0:01:37.271 *********** 2020-07-13T06:08:07.067 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "gid": 1222, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-07-13T06:08:07.122 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:07 +0000 (0:00:00.068) 0:01:37.339 *********** 2020-07-13T06:08:07.211 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-07-13T06:08:07.263 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:07 +0000 (0:00:00.140) 0:01:37.480 *********** 2020-07-13T06:08:07.284 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-07-13T06:08:07.326 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:07 +0000 (0:00:00.061) 0:01:37.542 *********** 2020-07-13T06:08:07.713 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1222, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-07-13T06:08:07.754 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:07 +0000 (0:00:00.427) 0:01:37.970 *********** 2020-07-13T06:08:07.800 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1222, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-07-13T06:08:07.838 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:07 +0000 (0:00:00.084) 0:01:38.055 *********** 2020-07-13T06:08:08.184 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-07-13T06:08:08.221 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:08 +0000 (0:00:00.382) 0:01:38.437 *********** 2020-07-13T06:08:08.255 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-07-13T06:08:08.303 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:08 +0000 (0:00:00.082) 0:01:38.520 *********** 2020-07-13T06:08:09.189 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-07-13T06:08:09.190 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"cache_update_time": 1594620381, "cache_updated": false, "changed": false} 2020-07-13T06:08:09.239 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:09 +0000 (0:00:00.935) 0:01:39.455 *********** 2020-07-13T06:08:09.260 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1594620377, "cache_updated": false, "changed": false} 2020-07-13T06:08:09.304 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:09 +0000 (0:00:00.065) 0:01:39.520 *********** 2020-07-13T06:08:09.431 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-07-13T06:08:09.432 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.042205", "end": "2020-07-13 06:08:09.625940", "rc": 0, "start": "2020-07-13 06:08:09.583735", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-07-13T06:08:09.488 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:09 +0000 (0:00:00.183) 0:01:39.704 *********** 2020-07-13T06:08:09.522 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.047982", "end": "2020-07-13 06:08:08.860205", "rc": 0, "start": "2020-07-13 06:08:08.812223", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-07-13T06:08:09.560 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:09 +0000 (0:00:00.072) 0:01:39.777 *********** 2020-07-13T06:08:09.571 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-07-13T06:08:09.573 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:09.612 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:09 +0000 (0:00:00.051) 0:01:39.828 *********** 2020-07-13T06:08:09.629 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:09.671 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:09 +0000 (0:00:00.058) 0:01:39.887 *********** 2020-07-13T06:08:09.681 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-07-13T06:08:09.683 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:09.730 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:09 +0000 (0:00:00.058) 0:01:39.946 *********** 2020-07-13T06:08:09.741 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:09.785 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:09 +0000 (0:00:00.055) 0:01:40.002 *********** 2020-07-13T06:08:10.617 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-07-13T06:08:10.619 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"cache_update_time": 1594620381, "cache_updated": false, "changed": false} 2020-07-13T06:08:10.658 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:10 +0000 (0:00:00.871) 0:01:40.874 *********** 2020-07-13T06:08:10.694 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1594620377, "cache_updated": false, "changed": false} 2020-07-13T06:08:10.702 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:10.744 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:10 +0000 (0:00:00.086) 0:01:40.960 *********** 2020-07-13T06:08:10.789 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:10 +0000 (0:00:00.044) 0:01:41.005 *********** 2020-07-13T06:08:10.822 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:10.862 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:10 +0000 (0:00:00.073) 0:01:41.078 *********** 2020-07-13T06:08:11.121 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-07-13T06:08:11.167 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:11 +0000 (0:00:00.304) 0:01:41.383 *********** 2020-07-13T06:08:11.198 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-07-13T06:08:11.200 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:11.239 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:11 +0000 (0:00:00.071) 0:01:41.455 *********** 2020-07-13T06:08:11.298 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-07-13T06:08:11.349 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:11 +0000 (0:00:00.109) 0:01:41.564 *********** 2020-07-13T06:08:11.393 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-07-13T06:08:11.394 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:11.448 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:11 +0000 (0:00:00.100) 0:01:41.664 *********** 2020-07-13T06:08:11.485 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-07-13T06:08:11.488 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-07-13 06:05:36 UTC", "ActiveEnterTimestampMonotonic": "17915006", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount sysinit.target systemd-tmpfiles-setup.service basic.target network.target system.slice systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-07-13 06:05:36 UTC", "AssertTimestampMonotonic": "17748168", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-07-13 06:05:36 UTC", "ConditionTimestampMonotonic": "17748168", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-07-13T06:08:11.490 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "883", "ExecMainStartTimestamp": "Mon 2020-07-13 06:05:36 UTC", "ExecMainStartTimestampMonotonic": "17914984", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-07-13 06:05:36 UTC", "InactiveExitTimestampMonotonic": "17748830", "InvocationID": "748012a059794d1e8590d2e89a01df74", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limit 2020-07-13T06:08:11.493 INFO:teuthology.task.ansible.out:RTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "883", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-07-13 06:05:36 UTC", "Sta 2020-07-13T06:08:11.494 INFO:teuthology.task.ansible.out:teChangeTimestampMonotonic": "17915006", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-07-13 06:05:36 UTC", "WatchdogTimestampMonotonic": "17915005", "WatchdogUSec": "0"}} 2020-07-13T06:08:11.568 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:11 +0000 (0:00:00.119) 0:01:41.784 *********** 2020-07-13T06:08:11.613 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi014.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-07-13T06:08:11.668 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:11 +0000 (0:00:00.099) 0:01:41.884 *********** 2020-07-13T06:08:11.702 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-13T06:08:11.704 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:11.704 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-07-13T06:08:11.706 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-07-13 06:05:39 UTC", "ActiveEnterTimestampMonotonic": "17450856", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target system.slice systemd-tmpfiles-setup.service -.mount systemd-journald.socket network.target basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-07-13 06:05:39 UTC", "AssertTimestampMonotonic": "17356064", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-07-13 06:05:39 UTC", "ConditionTimestampMonotonic": "17356064", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-07-13T06:08:11.708 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1085", "ExecMainStartTimestamp": "Mon 2020-07-13 06:05:39 UTC", "ExecMainStartTimestampMonotonic": "17450845", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-07-13 06:05:39 UTC", "InactiveExitTimestampMonotonic": "17356833", "InvocationID": "238ece0615bd4cc4a4e2da8a1344c142", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-07-13T06:08:11.709 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1085", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-07-13 06:05:39 UTC", "S 2020-07-13T06:08:11.712 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "17450856", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-07-13 06:05:39 UTC", "WatchdogTimestampMonotonic": "17450855", "WatchdogUSec": "0"}} 2020-07-13T06:08:11.749 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:11 +0000 (0:00:00.080) 0:01:41.965 *********** 2020-07-13T06:08:11.789 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:11 +0000 (0:00:00.040) 0:01:42.006 *********** 2020-07-13T06:08:11.801 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-07-13T06:08:11.803 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:11.848 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:11 +0000 (0:00:00.058) 0:01:42.064 *********** 2020-07-13T06:08:11.863 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-07-13T06:08:11.904 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:11 +0000 (0:00:00.055) 0:01:42.120 *********** 2020-07-13T06:08:11.966 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:11.998 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:11 +0000 (0:00:00.094) 0:01:42.214 *********** 2020-07-13T06:08:12.030 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-07-13T06:08:12.032 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:12.082 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:12 +0000 (0:00:00.083) 0:01:42.298 *********** 2020-07-13T06:08:12.750 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-07-13T06:08:12.752 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"cache_update_time": 1594620381, "cache_updated": false, "changed": false} 2020-07-13T06:08:12.798 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:12 +0000 (0:00:00.716) 0:01:43.014 *********** 2020-07-13T06:08:12.829 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-07-13T06:08:12.839 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:12.870 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:12 +0000 (0:00:00.072) 0:01:43.087 *********** 2020-07-13T06:08:12.990 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* ok: [smithi017.front.sepia.ceph.com] => {"cache_update_time": 1594620377, "cache_updated": false, "changed": false} 2020-07-13T06:08:13.032 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:13 +0000 (0:00:00.162) 0:01:43.249 *********** 2020-07-13T06:08:13.043 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-07-13T06:08:13.045 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "gid": 1222, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-07-13T06:08:13.100 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:13 +0000 (0:00:00.067) 0:01:43.316 *********** 2020-07-13T06:08:13.112 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-07-13T06:08:13.114 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:13.155 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:13 +0000 (0:00:00.054) 0:01:43.371 *********** 2020-07-13T06:08:13.468 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-07-13T06:08:13.470 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "gid": 1222, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-07-13T06:08:13.505 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:13 +0000 (0:00:00.350) 0:01:43.721 *********** 2020-07-13T06:08:13.611 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1222, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-07-13T06:08:13.655 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:13 +0000 (0:00:00.149) 0:01:43.871 *********** 2020-07-13T06:08:13.877 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1222, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-07-13T06:08:13.917 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:13 +0000 (0:00:00.261) 0:01:44.133 *********** 2020-07-13T06:08:14.665 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-07-13T06:08:14.667 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-07-13T06:08:14.708 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:14 +0000 (0:00:00.790) 0:01:44.924 *********** 2020-07-13T06:08:14.725 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-07-13T06:08:14.767 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:14 +0000 (0:00:00.059) 0:01:44.984 *********** 2020-07-13T06:08:15.270 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-07-13T06:08:15.272 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false} 2020-07-13T06:08:15.308 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:15 +0000 (0:00:00.540) 0:01:45.524 *********** 2020-07-13T06:08:15.329 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false} 2020-07-13T06:08:15.368 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:15 +0000 (0:00:00.060) 0:01:45.585 *********** 2020-07-13T06:08:15.379 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-07-13T06:08:15.381 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2020-07-13T06:08:15.421 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:15 +0000 (0:00:00.049) 0:01:45.634 *********** 2020-07-13T06:08:15.435 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2020-07-13T06:08:15.477 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:15 +0000 (0:00:00.059) 0:01:45.693 *********** 2020-07-13T06:08:15.588 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-07-13T06:08:15.590 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1594620352.1619353, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1594620352.1579354, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 105896, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1594620352.1579354, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "1235004745", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-07-13T06:08:15.634 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:15 +0000 (0:00:00.155) 0:01:45.849 *********** 2020-07-13T06:08:15.646 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1594620354.7122707, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1594620354.7082708, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 105896, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1594620354.7082708, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "1235004745", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-07-13T06:08:15.687 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:15 +0000 (0:00:00.053) 0:01:45.903 *********** 2020-07-13T06:08:15.950 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-07-13T06:08:15.951 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"changed": false, "msg": ""} ok: [smithi014.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-07-13T06:08:15.990 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:15 +0000 (0:00:00.303) 0:01:46.206 *********** 2020-07-13T06:08:16.034 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:16 +0000 (0:00:00.043) 0:01:46.250 *********** 2020-07-13T06:08:16.047 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-07-13T06:08:16.050 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-07-13T06:08:16.087 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:16 +0000 (0:00:00.053) 0:01:46.304 *********** 2020-07-13T06:08:16.098 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-07-13T06:08:16.138 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:16 +0000 (0:00:00.050) 0:01:46.354 *********** 2020-07-13T06:08:16.149 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-07-13T06:08:16.151 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:16.189 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:16 +0000 (0:00:00.051) 0:01:46.405 *********** 2020-07-13T06:08:16.201 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:16.241 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:16 +0000 (0:00:00.051) 0:01:46.457 *********** 2020-07-13T06:08:16.337 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-07-13T06:08:16.339 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-07-13T06:08:16.376 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:16 +0000 (0:00:00.135) 0:01:46.592 *********** 2020-07-13T06:08:16.397 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-07-13T06:08:16.437 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:16 +0000 (0:00:00.061) 0:01:46.654 *********** 2020-07-13T06:08:16.519 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-07-13T06:08:16.521 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-07-13T06:08:16.560 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:16 +0000 (0:00:00.122) 0:01:46.776 *********** 2020-07-13T06:08:16.592 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:16.645 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:16 +0000 (0:00:00.084) 0:01:46.861 *********** 2020-07-13T06:08:16.656 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-07-13T06:08:16.659 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-07-13T06:08:16.699 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:16 +0000 (0:00:00.054) 0:01:46.915 *********** 2020-07-13T06:08:16.708 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-07-13T06:08:16.710 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2020-07-13T06:08:16.711 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-07-13T06:08:16.749 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:16 +0000 (0:00:00.049) 0:01:46.965 *********** 2020-07-13T06:08:16.760 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-07-13T06:08:16.762 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:16.798 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:16 +0000 (0:00:00.049) 0:01:47.015 *********** 2020-07-13T06:08:16.809 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-07-13T06:08:16.811 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi014.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:16.811 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:16.812 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:16.812 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result 2020-07-13T06:08:16.814 INFO:teuthology.task.ansible.out: was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:16.886 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:16 +0000 (0:00:00.087) 0:01:47.102 *********** 2020-07-13T06:08:16.899 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-07-13T06:08:16.901 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-07-13T06:08:16.901 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2020-07-13T06:08:16.939 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:16 +0000 (0:00:00.053) 0:01:47.155 *********** 2020-07-13T06:08:16.949 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-13T06:08:16.951 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:16.993 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:16 +0000 (0:00:00.054) 0:01:47.210 *********** 2020-07-13T06:08:17.006 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-07-13T06:08:17.008 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:17.009 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi017.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result 2020-07-13T06:08:17.010 INFO:teuthology.task.ansible.out: was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:17.049 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:17 +0000 (0:00:00.054) 0:01:47.264 *********** 2020-07-13T06:08:17.059 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-07-13T06:08:17.061 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:17.100 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:17 +0000 (0:00:00.051) 0:01:47.316 *********** 2020-07-13T06:08:17.110 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-13T06:08:17.112 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:17.152 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:17 +0000 (0:00:00.052) 0:01:47.368 *********** 2020-07-13T06:08:17.162 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-07-13T06:08:17.165 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:17.203 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:17 +0000 (0:00:00.051) 0:01:47.419 *********** 2020-07-13T06:08:17.214 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-07-13T06:08:17.216 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:17.256 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:17 +0000 (0:00:00.052) 0:01:47.472 *********** 2020-07-13T06:08:17.273 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-07-13T06:08:17.274 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:17.323 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:17 +0000 (0:00:00.066) 0:01:47.539 *********** 2020-07-13T06:08:17.333 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-07-13T06:08:17.335 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:17.375 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:17 +0000 (0:00:00.052) 0:01:47.591 *********** 2020-07-13T06:08:17.385 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-07-13T06:08:17.389 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:17.430 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:17 +0000 (0:00:00.052) 0:01:47.644 *********** 2020-07-13T06:08:17.438 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-07-13T06:08:17.440 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:17.484 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:17 +0000 (0:00:00.056) 0:01:47.700 *********** 2020-07-13T06:08:17.496 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-07-13T06:08:17.498 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:17.540 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:17 +0000 (0:00:00.055) 0:01:47.756 *********** 2020-07-13T06:08:17.552 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-07-13T06:08:17.554 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:17.595 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:17 +0000 (0:00:00.054) 0:01:47.811 *********** 2020-07-13T06:08:17.605 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-07-13T06:08:17.607 INFO:teuthology.task.ansible.out:skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:17.646 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:17 +0000 (0:00:00.052) 0:01:47.863 *********** 2020-07-13T06:08:17.657 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-07-13T06:08:17.659 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:17.698 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:17 +0000 (0:00:00.051) 0:01:47.914 *********** 2020-07-13T06:08:17.708 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi014.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:17.749 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:17 +0000 (0:00:00.050) 0:01:47.965 *********** 2020-07-13T06:08:17.760 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-07-13T06:08:17.762 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:17.802 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:17 +0000 (0:00:00.052) 0:01:48.018 *********** 2020-07-13T06:08:17.863 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-07-13T06:08:17.865 INFO:teuthology.task.ansible.out:skipping: [smithi017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-13T06:08:17.913 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:17 +0000 (0:00:00.109) 0:01:48.128 *********** 2020-07-13T06:08:17.932 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi014.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011144", "end": "2020-07-13 06:08:18.103901", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-07-13 06:08:18.092757", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:08:18.091 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006136", "end": "2020-07-13 06:08:18.265068", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-07-13 06:08:18.258932", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} ok: [smithi017.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011007", "end": "2020-07-13 06:08:17.432029", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-07-13 06:08:17.421022", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:08:18.222 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005571", "end": "2020-07-13 06:08:18.408837", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-07-13 06:08:18.403266", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-07-13T06:08:18.234 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006085", "end": "2020-07-13 06:08:17.586643", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-07-13 06:08:17.580558", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:08:18.368 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005548", "end": "2020-07-13 06:08:17.712930", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-07-13 06:08:17.707382", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-07-13T06:08:18.401 INFO:teuthology.task.ansible.out:ok: [smithi014.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.024626", "end": "2020-07-13 06:08:18.566322", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-07-13 06:08:18.541696", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:08:18.441 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:18 +0000 (0:00:00.529) 0:01:48.657 *********** 2020-07-13T06:08:18.471 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-07-13T06:08:18.508 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:18 +0000 (0:00:00.067) 0:01:48.725 *********** 2020-07-13T06:08:18.529 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-07-13T06:08:18.531 INFO:teuthology.task.ansible.out:ok: [smithi017.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.026735", "end": "2020-07-13 06:08:17.869756", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-07-13 06:08:17.843021", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-13T06:08:18.572 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:18 +0000 (0:00:00.061) 0:01:48.786 *********** 2020-07-13T06:08:18.621 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-07-13T06:08:18.662 INFO:teuthology.task.ansible.out:Monday 13 July 2020 06:08:18 +0000 (0:00:00.092) 0:01:48.878 *********** 2020-07-13T06:08:18.671 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-07-13T06:08:18.673 INFO:teuthology.task.ansible.out:changed: [smithi014.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-07-13T06:08:18.807 INFO:teuthology.task.ansible.out:changed: [smithi017.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-07-13T06:08:18.913 INFO:teuthology.task.ansible.out: 2020-07-13T06:08:18.915 INFO:teuthology.task.ansible.out:PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-07-13T06:08:18.918 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** 2020-07-13T06:08:18.919 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2020-07-13T06:08:18.922 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2020-07-13T06:08:18.924 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi014.front.sepia.ceph.com : ok=108 changed=26 unreachable=0 failed=0 skipped=181 rescued=0 ignored=0 smithi017.front.sepia.ceph.com : ok=108 changed=26 unreachable=0 failed=0 skipped=181 rescued=0 ignored=0 Monday 13 July 2020 06:08:18 +0000 (0:00:00.260) 0:01:49.139 *********** =============================================================================== testnode : Ensure packages are not present. ----------------------------- 9.19s testnode : Install packages --------------------------------------------- 8.83s testnode : Zap all non-root disks --------------------------------------- 8.05s testnode : Add ssh pubkeys ---------------------------------------------- 5.87s common : Update apt cache ----------------------------------------------- 3.08s testnode : Update apt cache. -------------------------------------------- 2.89s testnode : Install apt keys --------------------------------------------- 2.29s Gathering Facts --------------------------------------------------------- 1.96s testnode : Create logical volume(s) ------------------------------------- 1.52s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.36s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.03s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 0.96s testnode : Ensure perl-doc is installed on apt systems. ----------------- 0.94s testnode : Set the system's hostname ------------------------------------ 0.91s testnode : Install python-apt ------------------------------------------- 0.91s testnode : Install non aarch64 packages --------------------------------- 0.90s testnode : Install ntp package on apt based systems. -------------------- 0.87s common : Install Kerberos Packages (Debian) ----------------------------- 0.85s Gathering Facts 2020-07-13T06:08:18.925 INFO:teuthology.task.ansible.out: --------------------------------------------------------- 0.84s testnode : Install packages via pip ------------------------------------- 0.79s 2020-07-13T06:08:19.299 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi017.front.sepia.ceph.com'), Remote(name='ubuntu@smithi014.front.sepia.ceph.com')] 2020-07-13T06:08:19.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-07-13T06:08:19.306 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:08:19.433 INFO:teuthology.orchestra.run.smithi017:> true 2020-07-13T06:08:19.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-07-13T06:08:19.628 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi014.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-13T06:08:19.698 INFO:teuthology.orchestra.run.smithi014:> true 2020-07-13T06:08:19.907 INFO:teuthology.run_tasks:Running task clock... 2020-07-13T06:08:19.920 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-07-13T06:08:19.921 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-07-13T06:08:19.921 INFO:teuthology.orchestra.run.smithi017:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-07-13T06:08:19.987 INFO:teuthology.orchestra.run.smithi017.stdout:13 Jul 06:08:19 ntpd[6563]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-07-13T06:08:19.987 INFO:teuthology.orchestra.run.smithi017.stdout:13 Jul 06:08:19 ntpd[6563]: Command line: ntpd -gq 2020-07-13T06:08:19.989 INFO:teuthology.orchestra.run.smithi017.stdout:13 Jul 06:08:19 ntpd[6563]: proto: precision = 0.117 usec (-23) 2020-07-13T06:08:19.989 INFO:teuthology.orchestra.run.smithi017.stdout:13 Jul 06:08:19 ntpd[6563]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-07-13T06:08:19.989 INFO:teuthology.orchestra.run.smithi017.stdout:13 Jul 06:08:19 ntpd[6563]: restrict ::: KOD does nothing without LIMITED. 2020-07-13T06:08:19.990 INFO:teuthology.orchestra.run.smithi017.stdout:13 Jul 06:08:19 ntpd[6563]: Listen and drop on 0 v6wildcard [::]:123 2020-07-13T06:08:19.990 INFO:teuthology.orchestra.run.smithi017.stdout:13 Jul 06:08:19 ntpd[6563]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-07-13T06:08:19.990 INFO:teuthology.orchestra.run.smithi017.stdout:13 Jul 06:08:19 ntpd[6563]: Listen normally on 2 lo 127.0.0.1:123 2020-07-13T06:08:19.991 INFO:teuthology.orchestra.run.smithi017.stdout:13 Jul 06:08:19 ntpd[6563]: Listen normally on 3 ens1f0 172.21.15.17:123 2020-07-13T06:08:19.991 INFO:teuthology.orchestra.run.smithi017.stdout:13 Jul 06:08:19 ntpd[6563]: Listen normally on 4 lo [::1]:123 2020-07-13T06:08:19.991 INFO:teuthology.orchestra.run.smithi017.stdout:13 Jul 06:08:19 ntpd[6563]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:fe59:d992%3]:123 2020-07-13T06:08:19.991 INFO:teuthology.orchestra.run.smithi017.stdout:13 Jul 06:08:19 ntpd[6563]: Listening on routing socket on fd #22 for interface updates 2020-07-13T06:08:19.992 INFO:teuthology.orchestra.run.smithi017.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-07-13T06:08:19.992 INFO:teuthology.orchestra.run.smithi017.stderr:restrict ::: KOD does nothing without LIMITED. 2020-07-13T06:08:23.988 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:23 ntpd[6563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:23.988 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:23 ntpd[6563]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:23.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:23 ntpd[6563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:23.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:23 ntpd[6563]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:24.988 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:24 ntpd[6563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:24.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:24 ntpd[6563]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:24.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:24 ntpd[6563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:24.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:24 ntpd[6563]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:24.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:24 ntpd[6563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:24.990 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:24 ntpd[6563]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:24.990 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:24 ntpd[6563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:24.990 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:24 ntpd[6563]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:24.990 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:24 ntpd[6563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:24.990 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:24 ntpd[6563]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:24.991 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:24 ntpd[6563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:24.991 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:24 ntpd[6563]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:25.987 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:25 ntpd[6563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:25.988 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:25 ntpd[6563]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:25.988 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:25 ntpd[6563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:25.988 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:25 ntpd[6563]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:26.988 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:26 ntpd[6563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:26.988 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:26 ntpd[6563]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:26.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:26 ntpd[6563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:26.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:26 ntpd[6563]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:26.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:26 ntpd[6563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:26.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:26 ntpd[6563]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:26.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:26 ntpd[6563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:26.990 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:26 ntpd[6563]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:26.990 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:26 ntpd[6563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:26.990 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:26 ntpd[6563]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:26.990 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:26 ntpd[6563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:26.990 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:26 ntpd[6563]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:27.987 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:27 ntpd[6563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:27.988 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:27 ntpd[6563]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:27.988 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:27 ntpd[6563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:27.988 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:27 ntpd[6563]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:28.988 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:28 ntpd[6563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:28.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:28 ntpd[6563]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:28.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:28 ntpd[6563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:28.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:28 ntpd[6563]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:28.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:28 ntpd[6563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:28.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:28 ntpd[6563]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:28.990 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:28 ntpd[6563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:28.990 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:28 ntpd[6563]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:28.990 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:28 ntpd[6563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:28.990 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:28 ntpd[6563]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:28.990 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:28 ntpd[6563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:28.991 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:28 ntpd[6563]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:29.988 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:29 ntpd[6563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:29.988 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:29 ntpd[6563]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:29.988 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:29 ntpd[6563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:29.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:29 ntpd[6563]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:29.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:29 ntpd[6563]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-07-13T06:08:29.989 INFO:teuthology.orchestra.run.smithi017.stderr:13 Jul 06:08:29 ntpd[6563]: can't open /var/log/ntpstats/loopstats.20200713: Permission denied 2020-07-13T06:08:29.989 INFO:teuthology.orchestra.run.smithi017.stdout:13 Jul 06:08:29 ntpd[6563]: ntpd: time set +0.616874 s 2020-07-13T06:08:29.990 INFO:teuthology.orchestra.run.smithi017.stdout:ntpd: time set +0.616874s 2020-07-13T06:08:30.041 INFO:teuthology.orchestra.run.smithi017.stdout: remote refid st t when poll reach delay offset jitter 2020-07-13T06:08:30.041 INFO:teuthology.orchestra.run.smithi017.stdout:============================================================================== 2020-07-13T06:08:30.041 INFO:teuthology.orchestra.run.smithi017.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-13T06:08:30.042 INFO:teuthology.orchestra.run.smithi017.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-13T06:08:30.042 INFO:teuthology.orchestra.run.smithi017.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-13T06:08:30.042 INFO:teuthology.orchestra.run.smithi017.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-13T06:08:30.043 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-07-13T06:08:30.044 INFO:teuthology.orchestra.run.smithi014:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-07-13T06:08:30.129 INFO:teuthology.orchestra.run.smithi014.stdout:13 Jul 06:08:30 ntpd[6497]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-07-13T06:08:30.129 INFO:teuthology.orchestra.run.smithi014.stdout:13 Jul 06:08:30 ntpd[6497]: Command line: ntpd -gq 2020-07-13T06:08:30.130 INFO:teuthology.orchestra.run.smithi014.stdout:13 Jul 06:08:30 ntpd[6497]: proto: precision = 0.127 usec (-23) 2020-07-13T06:08:30.131 INFO:teuthology.orchestra.run.smithi014.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-07-13T06:08:30.131 INFO:teuthology.orchestra.run.smithi014.stderr:restrict ::: KOD does nothing without LIMITED. 2020-07-13T06:08:30.131 INFO:teuthology.orchestra.run.smithi014.stdout:13 Jul 06:08:30 ntpd[6497]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-07-13T06:08:30.132 INFO:teuthology.orchestra.run.smithi014.stdout:13 Jul 06:08:30 ntpd[6497]: restrict ::: KOD does nothing without LIMITED. 2020-07-13T06:08:30.132 INFO:teuthology.orchestra.run.smithi014.stdout:13 Jul 06:08:30 ntpd[6497]: Listen and drop on 0 v6wildcard [::]:123 2020-07-13T06:08:30.132 INFO:teuthology.orchestra.run.smithi014.stdout:13 Jul 06:08:30 ntpd[6497]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-07-13T06:08:30.132 INFO:teuthology.orchestra.run.smithi014.stdout:13 Jul 06:08:30 ntpd[6497]: Listen normally on 2 lo 127.0.0.1:123 2020-07-13T06:08:30.132 INFO:teuthology.orchestra.run.smithi014.stdout:13 Jul 06:08:30 ntpd[6497]: Listen normally on 3 ens1f0 172.21.15.14:123 2020-07-13T06:08:30.133 INFO:teuthology.orchestra.run.smithi014.stdout:13 Jul 06:08:30 ntpd[6497]: Listen normally on 4 lo [::1]:123 2020-07-13T06:08:30.133 INFO:teuthology.orchestra.run.smithi014.stdout:13 Jul 06:08:30 ntpd[6497]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1250%3]:123 2020-07-13T06:08:30.134 INFO:teuthology.orchestra.run.smithi014.stdout:13 Jul 06:08:30 ntpd[6497]: Listening on routing socket on fd #22 for interface updates 2020-07-13T06:08:31.130 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:31 ntpd[6497]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:31.130 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:31 ntpd[6497]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:31.130 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:31 ntpd[6497]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:31.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:31 ntpd[6497]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:32.129 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:32 ntpd[6497]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:32.130 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:32 ntpd[6497]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:32.130 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:32 ntpd[6497]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:32.130 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:32 ntpd[6497]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:33.130 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:33 ntpd[6497]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:33.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:33 ntpd[6497]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:33.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:33 ntpd[6497]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:33.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:33 ntpd[6497]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:34.130 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:34 ntpd[6497]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:34.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:34 ntpd[6497]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:34.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:34 ntpd[6497]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:34.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:34 ntpd[6497]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:34.132 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:34 ntpd[6497]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:34.132 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:34 ntpd[6497]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:34.132 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:34 ntpd[6497]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:34.132 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:34 ntpd[6497]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:35.130 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:35 ntpd[6497]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:35.130 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:35 ntpd[6497]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:35.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:35 ntpd[6497]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:35.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:35 ntpd[6497]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:35.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:35 ntpd[6497]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:35.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:35 ntpd[6497]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:35.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:35 ntpd[6497]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:35.132 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:35 ntpd[6497]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:36.130 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:36 ntpd[6497]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:36.130 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:36 ntpd[6497]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:36.130 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:36 ntpd[6497]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:36.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:36 ntpd[6497]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:36.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:36 ntpd[6497]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:36.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:36 ntpd[6497]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:36.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:36 ntpd[6497]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:36.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:36 ntpd[6497]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:37.130 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:37 ntpd[6497]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:37.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:37 ntpd[6497]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:37.131 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:37 ntpd[6497]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:37.132 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:37 ntpd[6497]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:37.132 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:37 ntpd[6497]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-13T06:08:37.132 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:37 ntpd[6497]: can't open /var/log/ntpstats/rawstats.20200713: Permission denied 2020-07-13T06:08:37.132 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:37 ntpd[6497]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-13T06:08:37.132 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:37 ntpd[6497]: can't open /var/log/ntpstats/peerstats.20200713: Permission denied 2020-07-13T06:08:37.133 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:37 ntpd[6497]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-07-13T06:08:37.133 INFO:teuthology.orchestra.run.smithi014.stderr:13 Jul 06:08:37 ntpd[6497]: can't open /var/log/ntpstats/loopstats.20200713: Permission denied 2020-07-13T06:08:37.133 INFO:teuthology.orchestra.run.smithi014.stdout:13 Jul 06:08:37 ntpd[6497]: ntpd: time set -0.218341 s 2020-07-13T06:08:37.133 INFO:teuthology.orchestra.run.smithi014.stdout:ntpd: time set -0.218341s 2020-07-13T06:08:37.193 INFO:teuthology.orchestra.run.smithi014.stdout: remote refid st t when poll reach delay offset jitter 2020-07-13T06:08:37.194 INFO:teuthology.orchestra.run.smithi014.stdout:============================================================================== 2020-07-13T06:08:37.194 INFO:teuthology.orchestra.run.smithi014.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-13T06:08:37.194 INFO:teuthology.orchestra.run.smithi014.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-13T06:08:37.194 INFO:teuthology.orchestra.run.smithi014.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-13T06:08:37.195 INFO:teuthology.orchestra.run.smithi014.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-13T06:08:37.196 INFO:teuthology.run_tasks:Running task install... 2020-07-13T06:08:37.207 DEBUG:teuthology.task.install:project ceph 2020-07-13T06:08:37.208 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '1ac5c74502da6dc8ad965625df452eb42a846489'}} 2020-07-13T06:08:37.208 DEBUG:teuthology.task.install:config {'sha1': '1ac5c74502da6dc8ad965625df452eb42a846489'} 2020-07-13T06:08:37.209 INFO:teuthology.task.install:Using flavor: basic 2020-07-13T06:08:37.217 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-07-13T06:08:37.218 INFO:teuthology.task.install:extra packages: [] 2020-07-13T06:08:37.219 INFO:teuthology.orchestra.run.smithi017:> sudo lsb_release -is 2020-07-13T06:08:37.272 INFO:teuthology.orchestra.run.smithi017.stdout:Ubuntu 2020-07-13T06:08:37.278 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-07-13T06:08:37.279 INFO:teuthology.orchestra.run.smithi014:> sudo lsb_release -is 2020-07-13T06:08:37.279 INFO:teuthology.orchestra.run.smithi017:> sudo apt-key list | grep Ceph 2020-07-13T06:08:37.333 INFO:teuthology.orchestra.run.smithi014.stdout:Ubuntu 2020-07-13T06:08:37.339 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-07-13T06:08:37.340 INFO:teuthology.orchestra.run.smithi014:> sudo apt-key list | grep Ceph 2020-07-13T06:08:37.377 INFO:teuthology.orchestra.run.smithi017.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-07-13T06:08:37.432 INFO:teuthology.orchestra.run.smithi014.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-07-13T06:08:37.439 INFO:teuthology.orchestra.run.smithi017.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-07-13T06:08:37.439 INFO:teuthology.orchestra.run.smithi017.stdout:uid [ unknown] Ceph.com (release key) 2020-07-13T06:08:37.440 INFO:teuthology.orchestra.run.smithi017:> sudo lsb_release -is 2020-07-13T06:08:37.486 INFO:teuthology.orchestra.run.smithi014.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-07-13T06:08:37.486 INFO:teuthology.orchestra.run.smithi014.stdout:uid [ unknown] Ceph.com (release key) 2020-07-13T06:08:37.487 INFO:teuthology.orchestra.run.smithi014:> sudo lsb_release -is 2020-07-13T06:08:37.491 INFO:teuthology.orchestra.run.smithi017.stdout:Ubuntu 2020-07-13T06:08:37.497 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-07-13T06:08:37.497 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-07-13T06:08:37.498 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=1ac5c74502da6dc8ad965625df452eb42a846489 2020-07-13T06:08:37.543 INFO:teuthology.orchestra.run.smithi014.stdout:Ubuntu 2020-07-13T06:08:37.544 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-07-13T06:08:37.544 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-07-13T06:08:37.545 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=1ac5c74502da6dc8ad965625df452eb42a846489 2020-07-13T06:08:37.676 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default/ 2020-07-13T06:08:37.677 INFO:teuthology.task.install.deb:Package version is 16.0.0-3402-g1ac5c74-1bionic 2020-07-13T06:08:37.723 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default/ 2020-07-13T06:08:37.724 INFO:teuthology.task.install.deb:Package version is 16.0.0-3402-g1ac5c74-1bionic 2020-07-13T06:08:37.816 INFO:teuthology.orchestra.run.smithi017:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2020-07-13T06:08:37.862 INFO:teuthology.orchestra.run.smithi014:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2020-07-13T06:08:37.864 INFO:teuthology.orchestra.run.smithi017:> sudo apt-get update 2020-07-13T06:08:37.914 INFO:teuthology.orchestra.run.smithi014:> sudo apt-get update 2020-07-13T06:08:38.161 INFO:teuthology.orchestra.run.smithi017.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic InRelease 2020-07-13T06:08:38.176 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-07-13T06:08:38.177 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-07-13T06:08:38.184 INFO:teuthology.orchestra.run.smithi017.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-07-13T06:08:38.186 INFO:teuthology.orchestra.run.smithi014.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic InRelease 2020-07-13T06:08:38.209 INFO:teuthology.orchestra.run.smithi017.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic Release.gpg 2020-07-13T06:08:38.210 INFO:teuthology.orchestra.run.smithi014.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-07-13T06:08:38.232 INFO:teuthology.orchestra.run.smithi017.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,083 B] 2020-07-13T06:08:38.234 INFO:teuthology.orchestra.run.smithi014.stdout:Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-07-13T06:08:38.234 INFO:teuthology.orchestra.run.smithi014.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-07-13T06:08:38.236 INFO:teuthology.orchestra.run.smithi014.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic Release.gpg 2020-07-13T06:08:38.249 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-07-13T06:08:38.259 INFO:teuthology.orchestra.run.smithi014.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.9 kB] 2020-07-13T06:08:38.272 INFO:teuthology.orchestra.run.smithi017.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.9 kB] 2020-07-13T06:08:38.297 INFO:teuthology.orchestra.run.smithi014.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,083 B] 2020-07-13T06:08:38.303 INFO:teuthology.orchestra.run.smithi014.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-07-13T06:08:38.327 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-07-13T06:08:38.383 INFO:teuthology.orchestra.run.smithi014.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-07-13T06:08:38.798 INFO:teuthology.orchestra.run.smithi017.stdout:Fetched 26.6 kB in 1s (36.6 kB/s) 2020-07-13T06:08:38.851 INFO:teuthology.orchestra.run.smithi014.stdout:Fetched 26.6 kB in 1s (36.8 kB/s) 2020-07-13T06:08:40.035 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2020-07-13T06:08:40.083 INFO:teuthology.orchestra.run.smithi017:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-3402-g1ac5c74-1bionic cephadm=16.0.0-3402-g1ac5c74-1bionic ceph-mds=16.0.0-3402-g1ac5c74-1bionic ceph-mgr=16.0.0-3402-g1ac5c74-1bionic ceph-common=16.0.0-3402-g1ac5c74-1bionic ceph-fuse=16.0.0-3402-g1ac5c74-1bionic ceph-test=16.0.0-3402-g1ac5c74-1bionic radosgw=16.0.0-3402-g1ac5c74-1bionic python3-rados=16.0.0-3402-g1ac5c74-1bionic python3-rgw=16.0.0-3402-g1ac5c74-1bionic python3-cephfs=16.0.0-3402-g1ac5c74-1bionic python3-rbd=16.0.0-3402-g1ac5c74-1bionic libcephfs2=16.0.0-3402-g1ac5c74-1bionic libcephfs-dev=16.0.0-3402-g1ac5c74-1bionic librados2=16.0.0-3402-g1ac5c74-1bionic librbd1=16.0.0-3402-g1ac5c74-1bionic rbd-fuse=16.0.0-3402-g1ac5c74-1bionic 2020-07-13T06:08:40.087 INFO:teuthology.orchestra.run.smithi014.stdout:Reading package lists... 2020-07-13T06:08:40.133 INFO:teuthology.orchestra.run.smithi014:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-3402-g1ac5c74-1bionic cephadm=16.0.0-3402-g1ac5c74-1bionic ceph-mds=16.0.0-3402-g1ac5c74-1bionic ceph-mgr=16.0.0-3402-g1ac5c74-1bionic ceph-common=16.0.0-3402-g1ac5c74-1bionic ceph-fuse=16.0.0-3402-g1ac5c74-1bionic ceph-test=16.0.0-3402-g1ac5c74-1bionic radosgw=16.0.0-3402-g1ac5c74-1bionic python3-rados=16.0.0-3402-g1ac5c74-1bionic python3-rgw=16.0.0-3402-g1ac5c74-1bionic python3-cephfs=16.0.0-3402-g1ac5c74-1bionic python3-rbd=16.0.0-3402-g1ac5c74-1bionic libcephfs2=16.0.0-3402-g1ac5c74-1bionic libcephfs-dev=16.0.0-3402-g1ac5c74-1bionic librados2=16.0.0-3402-g1ac5c74-1bionic librbd1=16.0.0-3402-g1ac5c74-1bionic rbd-fuse=16.0.0-3402-g1ac5c74-1bionic 2020-07-13T06:08:40.167 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2020-07-13T06:08:40.215 INFO:teuthology.orchestra.run.smithi014.stdout:Reading package lists... 2020-07-13T06:08:40.408 INFO:teuthology.orchestra.run.smithi017.stdout:Building dependency tree... 2020-07-13T06:08:40.408 INFO:teuthology.orchestra.run.smithi017.stdout:Reading state information... 2020-07-13T06:08:40.446 INFO:teuthology.orchestra.run.smithi014.stdout:Building dependency tree... 2020-07-13T06:08:40.447 INFO:teuthology.orchestra.run.smithi014.stdout:Reading state information... 2020-07-13T06:08:40.596 INFO:teuthology.orchestra.run.smithi017.stdout:The following additional packages will be installed: 2020-07-13T06:08:40.597 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-07-13T06:08:40.597 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-07-13T06:08:40.598 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2020-07-13T06:08:40.598 INFO:teuthology.orchestra.run.smithi017.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2020-07-13T06:08:40.598 INFO:teuthology.orchestra.run.smithi017.stdout: libsnappy1v5 libxslt1.1 python-attr python-funcsigs python-pastedeploy-tpl 2020-07-13T06:08:40.599 INFO:teuthology.orchestra.run.smithi017.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-07-13T06:08:40.599 INFO:teuthology.orchestra.run.smithi017.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-07-13T06:08:40.599 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-07-13T06:08:40.599 INFO:teuthology.orchestra.run.smithi017.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-07-13T06:08:40.600 INFO:teuthology.orchestra.run.smithi017.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2020-07-13T06:08:40.600 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2020-07-13T06:08:40.600 INFO:teuthology.orchestra.run.smithi017.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2020-07-13T06:08:40.600 INFO:teuthology.orchestra.run.smithi017.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2020-07-13T06:08:40.600 INFO:teuthology.orchestra.run.smithi017.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2020-07-13T06:08:40.601 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-07-13T06:08:40.601 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-07-13T06:08:40.601 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pytest python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-07-13T06:08:40.601 INFO:teuthology.orchestra.run.smithi017.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-07-13T06:08:40.601 INFO:teuthology.orchestra.run.smithi017.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-07-13T06:08:40.602 INFO:teuthology.orchestra.run.smithi017.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-07-13T06:08:40.602 INFO:teuthology.orchestra.run.smithi017.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-07-13T06:08:40.602 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-07-13T06:08:40.603 INFO:teuthology.orchestra.run.smithi017.stdout:Suggested packages: 2020-07-13T06:08:40.603 INFO:teuthology.orchestra.run.smithi017.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-07-13T06:08:40.603 INFO:teuthology.orchestra.run.smithi017.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-07-13T06:08:40.603 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-07-13T06:08:40.603 INFO:teuthology.orchestra.run.smithi017.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-07-13T06:08:40.604 INFO:teuthology.orchestra.run.smithi017.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-07-13T06:08:40.604 INFO:teuthology.orchestra.run.smithi017.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-07-13T06:08:40.604 INFO:teuthology.orchestra.run.smithi017.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-07-13T06:08:40.604 INFO:teuthology.orchestra.run.smithi017.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-07-13T06:08:40.604 INFO:teuthology.orchestra.run.smithi017.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-07-13T06:08:40.605 INFO:teuthology.orchestra.run.smithi017.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-07-13T06:08:40.605 INFO:teuthology.orchestra.run.smithi017.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-07-13T06:08:40.605 INFO:teuthology.orchestra.run.smithi017.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-07-13T06:08:40.606 INFO:teuthology.orchestra.run.smithi017.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-07-13T06:08:40.606 INFO:teuthology.orchestra.run.smithi017.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-07-13T06:08:40.606 INFO:teuthology.orchestra.run.smithi017.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-07-13T06:08:40.622 INFO:teuthology.orchestra.run.smithi014.stdout:The following additional packages will be installed: 2020-07-13T06:08:40.623 INFO:teuthology.orchestra.run.smithi014.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-07-13T06:08:40.623 INFO:teuthology.orchestra.run.smithi014.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-07-13T06:08:40.624 INFO:teuthology.orchestra.run.smithi014.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2020-07-13T06:08:40.624 INFO:teuthology.orchestra.run.smithi014.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2020-07-13T06:08:40.624 INFO:teuthology.orchestra.run.smithi014.stdout: libsnappy1v5 libxslt1.1 python-attr python-funcsigs python-pastedeploy-tpl 2020-07-13T06:08:40.624 INFO:teuthology.orchestra.run.smithi014.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-07-13T06:08:40.625 INFO:teuthology.orchestra.run.smithi014.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-07-13T06:08:40.625 INFO:teuthology.orchestra.run.smithi014.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-07-13T06:08:40.625 INFO:teuthology.orchestra.run.smithi014.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-07-13T06:08:40.625 INFO:teuthology.orchestra.run.smithi014.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2020-07-13T06:08:40.625 INFO:teuthology.orchestra.run.smithi014.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2020-07-13T06:08:40.626 INFO:teuthology.orchestra.run.smithi014.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2020-07-13T06:08:40.626 INFO:teuthology.orchestra.run.smithi014.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2020-07-13T06:08:40.626 INFO:teuthology.orchestra.run.smithi014.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2020-07-13T06:08:40.627 INFO:teuthology.orchestra.run.smithi014.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-07-13T06:08:40.627 INFO:teuthology.orchestra.run.smithi014.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-07-13T06:08:40.627 INFO:teuthology.orchestra.run.smithi014.stdout: python3-pytest python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-07-13T06:08:40.627 INFO:teuthology.orchestra.run.smithi014.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-07-13T06:08:40.627 INFO:teuthology.orchestra.run.smithi014.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-07-13T06:08:40.628 INFO:teuthology.orchestra.run.smithi014.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-07-13T06:08:40.628 INFO:teuthology.orchestra.run.smithi014.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-07-13T06:08:40.628 INFO:teuthology.orchestra.run.smithi014.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-07-13T06:08:40.628 INFO:teuthology.orchestra.run.smithi014.stdout:Suggested packages: 2020-07-13T06:08:40.629 INFO:teuthology.orchestra.run.smithi014.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-07-13T06:08:40.629 INFO:teuthology.orchestra.run.smithi014.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-07-13T06:08:40.629 INFO:teuthology.orchestra.run.smithi014.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-07-13T06:08:40.629 INFO:teuthology.orchestra.run.smithi014.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-07-13T06:08:40.630 INFO:teuthology.orchestra.run.smithi014.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-07-13T06:08:40.630 INFO:teuthology.orchestra.run.smithi014.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-07-13T06:08:40.630 INFO:teuthology.orchestra.run.smithi014.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-07-13T06:08:40.630 INFO:teuthology.orchestra.run.smithi014.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-07-13T06:08:40.630 INFO:teuthology.orchestra.run.smithi014.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-07-13T06:08:40.630 INFO:teuthology.orchestra.run.smithi014.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-07-13T06:08:40.631 INFO:teuthology.orchestra.run.smithi014.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-07-13T06:08:40.631 INFO:teuthology.orchestra.run.smithi014.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-07-13T06:08:40.631 INFO:teuthology.orchestra.run.smithi014.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-07-13T06:08:40.631 INFO:teuthology.orchestra.run.smithi014.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-07-13T06:08:40.631 INFO:teuthology.orchestra.run.smithi014.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-07-13T06:08:40.708 INFO:teuthology.orchestra.run.smithi017.stdout:The following NEW packages will be installed: 2020-07-13T06:08:40.708 INFO:teuthology.orchestra.run.smithi017.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-07-13T06:08:40.709 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-07-13T06:08:40.709 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-07-13T06:08:40.710 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2020-07-13T06:08:40.710 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-07-13T06:08:40.711 INFO:teuthology.orchestra.run.smithi017.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-07-13T06:08:40.711 INFO:teuthology.orchestra.run.smithi017.stdout: python-funcsigs python-pastedeploy-tpl python-pluggy python-py python-pytest 2020-07-13T06:08:40.711 INFO:teuthology.orchestra.run.smithi017.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-07-13T06:08:40.711 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2020-07-13T06:08:40.711 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-07-13T06:08:40.712 INFO:teuthology.orchestra.run.smithi017.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-07-13T06:08:40.712 INFO:teuthology.orchestra.run.smithi017.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2020-07-13T06:08:40.712 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2020-07-13T06:08:40.712 INFO:teuthology.orchestra.run.smithi017.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2020-07-13T06:08:40.712 INFO:teuthology.orchestra.run.smithi017.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2020-07-13T06:08:40.713 INFO:teuthology.orchestra.run.smithi017.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2020-07-13T06:08:40.713 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-07-13T06:08:40.713 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-07-13T06:08:40.713 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2020-07-13T06:08:40.713 INFO:teuthology.orchestra.run.smithi017.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2020-07-13T06:08:40.714 INFO:teuthology.orchestra.run.smithi017.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2020-07-13T06:08:40.714 INFO:teuthology.orchestra.run.smithi017.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2020-07-13T06:08:40.714 INFO:teuthology.orchestra.run.smithi017.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2020-07-13T06:08:40.714 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2020-07-13T06:08:40.714 INFO:teuthology.orchestra.run.smithi017.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-07-13T06:08:40.715 INFO:teuthology.orchestra.run.smithi017.stdout:The following packages will be upgraded: 2020-07-13T06:08:40.715 INFO:teuthology.orchestra.run.smithi017.stdout: librados2 librbd1 2020-07-13T06:08:40.733 INFO:teuthology.orchestra.run.smithi014.stdout:The following NEW packages will be installed: 2020-07-13T06:08:40.733 INFO:teuthology.orchestra.run.smithi014.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-07-13T06:08:40.733 INFO:teuthology.orchestra.run.smithi014.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-07-13T06:08:40.734 INFO:teuthology.orchestra.run.smithi014.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-07-13T06:08:40.735 INFO:teuthology.orchestra.run.smithi014.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2020-07-13T06:08:40.735 INFO:teuthology.orchestra.run.smithi014.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-07-13T06:08:40.735 INFO:teuthology.orchestra.run.smithi014.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-07-13T06:08:40.735 INFO:teuthology.orchestra.run.smithi014.stdout: python-funcsigs python-pastedeploy-tpl python-pluggy python-py python-pytest 2020-07-13T06:08:40.736 INFO:teuthology.orchestra.run.smithi014.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-07-13T06:08:40.736 INFO:teuthology.orchestra.run.smithi014.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2020-07-13T06:08:40.736 INFO:teuthology.orchestra.run.smithi014.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-07-13T06:08:40.736 INFO:teuthology.orchestra.run.smithi014.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-07-13T06:08:40.736 INFO:teuthology.orchestra.run.smithi014.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2020-07-13T06:08:40.737 INFO:teuthology.orchestra.run.smithi014.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2020-07-13T06:08:40.737 INFO:teuthology.orchestra.run.smithi014.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2020-07-13T06:08:40.737 INFO:teuthology.orchestra.run.smithi014.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2020-07-13T06:08:40.737 INFO:teuthology.orchestra.run.smithi014.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2020-07-13T06:08:40.737 INFO:teuthology.orchestra.run.smithi014.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-07-13T06:08:40.738 INFO:teuthology.orchestra.run.smithi014.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-07-13T06:08:40.738 INFO:teuthology.orchestra.run.smithi014.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2020-07-13T06:08:40.738 INFO:teuthology.orchestra.run.smithi014.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2020-07-13T06:08:40.739 INFO:teuthology.orchestra.run.smithi014.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2020-07-13T06:08:40.739 INFO:teuthology.orchestra.run.smithi014.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2020-07-13T06:08:40.739 INFO:teuthology.orchestra.run.smithi014.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2020-07-13T06:08:40.739 INFO:teuthology.orchestra.run.smithi014.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2020-07-13T06:08:40.739 INFO:teuthology.orchestra.run.smithi014.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-07-13T06:08:40.740 INFO:teuthology.orchestra.run.smithi014.stdout:The following packages will be upgraded: 2020-07-13T06:08:40.740 INFO:teuthology.orchestra.run.smithi014.stdout: librados2 librbd1 2020-07-13T06:08:40.874 INFO:teuthology.orchestra.run.smithi017.stdout:2 upgraded, 110 newly installed, 0 to remove and 0 not upgraded. 2020-07-13T06:08:40.875 INFO:teuthology.orchestra.run.smithi017.stdout:Need to get 164 MB of archives. 2020-07-13T06:08:40.875 INFO:teuthology.orchestra.run.smithi017.stdout:After this operation, 765 MB of additional disk space will be used. 2020-07-13T06:08:40.875 INFO:teuthology.orchestra.run.smithi017.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-3402-g1ac5c74-1bionic [1,679 kB] 2020-07-13T06:08:40.899 INFO:teuthology.orchestra.run.smithi014.stdout:2 upgraded, 110 newly installed, 0 to remove and 0 not upgraded. 2020-07-13T06:08:40.899 INFO:teuthology.orchestra.run.smithi014.stdout:Need to get 164 MB of archives. 2020-07-13T06:08:40.899 INFO:teuthology.orchestra.run.smithi014.stdout:After this operation, 765 MB of additional disk space will be used. 2020-07-13T06:08:40.899 INFO:teuthology.orchestra.run.smithi014.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-3402-g1ac5c74-1bionic [1,679 kB] 2020-07-13T06:08:40.903 INFO:teuthology.orchestra.run.smithi017.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-07-13T06:08:40.923 INFO:teuthology.orchestra.run.smithi014.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-07-13T06:08:40.985 INFO:teuthology.orchestra.run.smithi017.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-07-13T06:08:41.002 INFO:teuthology.orchestra.run.smithi014.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-07-13T06:08:41.053 INFO:teuthology.orchestra.run.smithi017.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-3402-g1ac5c74-1bionic [3,146 kB] 2020-07-13T06:08:41.090 INFO:teuthology.orchestra.run.smithi014.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-3402-g1ac5c74-1bionic [3,146 kB] 2020-07-13T06:08:41.224 INFO:teuthology.orchestra.run.smithi017.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-3402-g1ac5c74-1bionic [460 kB] 2020-07-13T06:08:41.239 INFO:teuthology.orchestra.run.smithi017.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-07-13T06:08:41.247 INFO:teuthology.orchestra.run.smithi017.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-07-13T06:08:41.252 INFO:teuthology.orchestra.run.smithi014.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-07-13T06:08:41.259 INFO:teuthology.orchestra.run.smithi017.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-07-13T06:08:41.260 INFO:teuthology.orchestra.run.smithi017.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-3402-g1ac5c74-1bionic [345 kB] 2020-07-13T06:08:41.264 INFO:teuthology.orchestra.run.smithi014.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-07-13T06:08:41.270 INFO:teuthology.orchestra.run.smithi017.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-3402-g1ac5c74-1bionic [20.6 kB] 2020-07-13T06:08:41.270 INFO:teuthology.orchestra.run.smithi017.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-3402-g1ac5c74-1bionic [168 kB] 2020-07-13T06:08:41.276 INFO:teuthology.orchestra.run.smithi017.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-3402-g1ac5c74-1bionic [25.6 kB] 2020-07-13T06:08:41.277 INFO:teuthology.orchestra.run.smithi017.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-3402-g1ac5c74-1bionic [312 kB] 2020-07-13T06:08:41.282 INFO:teuthology.orchestra.run.smithi014.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-07-13T06:08:41.293 INFO:teuthology.orchestra.run.smithi014.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-3402-g1ac5c74-1bionic [460 kB] 2020-07-13T06:08:41.309 INFO:teuthology.orchestra.run.smithi017.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-3402-g1ac5c74-1bionic [3,005 kB] 2020-07-13T06:08:41.332 INFO:teuthology.orchestra.run.smithi014.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-3402-g1ac5c74-1bionic [345 kB] 2020-07-13T06:08:41.343 INFO:teuthology.orchestra.run.smithi014.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-3402-g1ac5c74-1bionic [20.6 kB] 2020-07-13T06:08:41.344 INFO:teuthology.orchestra.run.smithi014.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-3402-g1ac5c74-1bionic [168 kB] 2020-07-13T06:08:41.365 INFO:teuthology.orchestra.run.smithi017.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-07-13T06:08:41.368 INFO:teuthology.orchestra.run.smithi017.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-07-13T06:08:41.375 INFO:teuthology.orchestra.run.smithi014.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-3402-g1ac5c74-1bionic [25.6 kB] 2020-07-13T06:08:41.375 INFO:teuthology.orchestra.run.smithi014.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-3402-g1ac5c74-1bionic [312 kB] 2020-07-13T06:08:41.376 INFO:teuthology.orchestra.run.smithi014.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-07-13T06:08:41.379 INFO:teuthology.orchestra.run.smithi014.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-07-13T06:08:41.405 INFO:teuthology.orchestra.run.smithi017.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-07-13T06:08:41.407 INFO:teuthology.orchestra.run.smithi014.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-07-13T06:08:41.411 INFO:teuthology.orchestra.run.smithi017.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-07-13T06:08:41.413 INFO:teuthology.orchestra.run.smithi014.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-07-13T06:08:41.420 INFO:teuthology.orchestra.run.smithi017.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-07-13T06:08:41.423 INFO:teuthology.orchestra.run.smithi014.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-07-13T06:08:41.432 INFO:teuthology.orchestra.run.smithi017.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-07-13T06:08:41.433 INFO:teuthology.orchestra.run.smithi014.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-07-13T06:08:41.441 INFO:teuthology.orchestra.run.smithi014.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB] 2020-07-13T06:08:41.442 INFO:teuthology.orchestra.run.smithi017.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB] 2020-07-13T06:08:41.447 INFO:teuthology.orchestra.run.smithi014.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-3402-g1ac5c74-1bionic [3,005 kB] 2020-07-13T06:08:41.465 INFO:teuthology.orchestra.run.smithi014.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-07-13T06:08:41.466 INFO:teuthology.orchestra.run.smithi017.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-07-13T06:08:41.469 INFO:teuthology.orchestra.run.smithi014.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-07-13T06:08:41.472 INFO:teuthology.orchestra.run.smithi017.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-07-13T06:08:41.486 INFO:teuthology.orchestra.run.smithi017.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-07-13T06:08:41.488 INFO:teuthology.orchestra.run.smithi014.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-07-13T06:08:41.491 INFO:teuthology.orchestra.run.smithi017.stdout:Get:25 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-3402-g1ac5c74-1bionic [99.2 kB] 2020-07-13T06:08:41.496 INFO:teuthology.orchestra.run.smithi017.stdout:Get:26 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-3402-g1ac5c74-1bionic [413 kB] 2020-07-13T06:08:41.498 INFO:teuthology.orchestra.run.smithi017.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-07-13T06:08:41.499 INFO:teuthology.orchestra.run.smithi014.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-07-13T06:08:41.500 INFO:teuthology.orchestra.run.smithi014.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-07-13T06:08:41.507 INFO:teuthology.orchestra.run.smithi017.stdout:Get:28 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-3402-g1ac5c74-1bionic [19.7 MB] 2020-07-13T06:08:41.566 INFO:teuthology.orchestra.run.smithi014.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-07-13T06:08:41.567 INFO:teuthology.orchestra.run.smithi017.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-07-13T06:08:41.567 INFO:teuthology.orchestra.run.smithi014.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-07-13T06:08:41.568 INFO:teuthology.orchestra.run.smithi017.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-07-13T06:08:41.568 INFO:teuthology.orchestra.run.smithi017.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-07-13T06:08:41.569 INFO:teuthology.orchestra.run.smithi014.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-07-13T06:08:41.570 INFO:teuthology.orchestra.run.smithi014.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-07-13T06:08:41.571 INFO:teuthology.orchestra.run.smithi017.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-07-13T06:08:41.572 INFO:teuthology.orchestra.run.smithi014.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-07-13T06:08:41.573 INFO:teuthology.orchestra.run.smithi017.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-07-13T06:08:41.573 INFO:teuthology.orchestra.run.smithi017.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-07-13T06:08:41.575 INFO:teuthology.orchestra.run.smithi014.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-07-13T06:08:41.575 INFO:teuthology.orchestra.run.smithi017.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-07-13T06:08:41.576 INFO:teuthology.orchestra.run.smithi014.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-07-13T06:08:41.578 INFO:teuthology.orchestra.run.smithi017.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-07-13T06:08:41.644 INFO:teuthology.orchestra.run.smithi014.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-07-13T06:08:41.645 INFO:teuthology.orchestra.run.smithi014.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-07-13T06:08:41.647 INFO:teuthology.orchestra.run.smithi017.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-07-13T06:08:41.648 INFO:teuthology.orchestra.run.smithi017.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-07-13T06:08:41.662 INFO:teuthology.orchestra.run.smithi014.stdout:Get:36 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-3402-g1ac5c74-1bionic [99.2 kB] 2020-07-13T06:08:41.665 INFO:teuthology.orchestra.run.smithi014.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-07-13T06:08:41.668 INFO:teuthology.orchestra.run.smithi014.stdout:Get:38 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-3402-g1ac5c74-1bionic [413 kB] 2020-07-13T06:08:41.718 INFO:teuthology.orchestra.run.smithi014.stdout:Get:39 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-3402-g1ac5c74-1bionic [19.7 MB] 2020-07-13T06:08:41.722 INFO:teuthology.orchestra.run.smithi014.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-07-13T06:08:41.724 INFO:teuthology.orchestra.run.smithi014.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-07-13T06:08:41.725 INFO:teuthology.orchestra.run.smithi014.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-07-13T06:08:41.728 INFO:teuthology.orchestra.run.smithi017.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-07-13T06:08:41.728 INFO:teuthology.orchestra.run.smithi014.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-07-13T06:08:41.729 INFO:teuthology.orchestra.run.smithi017.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-07-13T06:08:41.729 INFO:teuthology.orchestra.run.smithi017.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-07-13T06:08:41.730 INFO:teuthology.orchestra.run.smithi017.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-07-13T06:08:41.734 INFO:teuthology.orchestra.run.smithi017.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-07-13T06:08:41.735 INFO:teuthology.orchestra.run.smithi014.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-07-13T06:08:41.739 INFO:teuthology.orchestra.run.smithi014.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-07-13T06:08:41.740 INFO:teuthology.orchestra.run.smithi014.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-07-13T06:08:41.742 INFO:teuthology.orchestra.run.smithi017.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-07-13T06:08:41.746 INFO:teuthology.orchestra.run.smithi017.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-07-13T06:08:41.747 INFO:teuthology.orchestra.run.smithi017.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-07-13T06:08:41.801 INFO:teuthology.orchestra.run.smithi014.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-07-13T06:08:41.808 INFO:teuthology.orchestra.run.smithi017.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-07-13T06:08:41.885 INFO:teuthology.orchestra.run.smithi014.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-07-13T06:08:41.885 INFO:teuthology.orchestra.run.smithi014.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-07-13T06:08:41.898 INFO:teuthology.orchestra.run.smithi017.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-07-13T06:08:41.899 INFO:teuthology.orchestra.run.smithi017.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-07-13T06:08:42.155 INFO:teuthology.orchestra.run.smithi014.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-07-13T06:08:42.187 INFO:teuthology.orchestra.run.smithi017.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-07-13T06:08:42.199 INFO:teuthology.orchestra.run.smithi014.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-07-13T06:08:42.200 INFO:teuthology.orchestra.run.smithi014.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-07-13T06:08:42.226 INFO:teuthology.orchestra.run.smithi017.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-07-13T06:08:42.228 INFO:teuthology.orchestra.run.smithi017.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-07-13T06:08:42.234 INFO:teuthology.orchestra.run.smithi014.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-07-13T06:08:42.234 INFO:teuthology.orchestra.run.smithi014.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-07-13T06:08:42.235 INFO:teuthology.orchestra.run.smithi014.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-07-13T06:08:42.235 INFO:teuthology.orchestra.run.smithi014.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-07-13T06:08:42.236 INFO:teuthology.orchestra.run.smithi014.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-07-13T06:08:42.236 INFO:teuthology.orchestra.run.smithi014.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-07-13T06:08:42.236 INFO:teuthology.orchestra.run.smithi014.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-07-13T06:08:42.270 INFO:teuthology.orchestra.run.smithi017.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-07-13T06:08:42.271 INFO:teuthology.orchestra.run.smithi017.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-07-13T06:08:42.271 INFO:teuthology.orchestra.run.smithi017.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-07-13T06:08:42.271 INFO:teuthology.orchestra.run.smithi017.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-07-13T06:08:42.273 INFO:teuthology.orchestra.run.smithi017.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-07-13T06:08:42.274 INFO:teuthology.orchestra.run.smithi017.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-07-13T06:08:42.275 INFO:teuthology.orchestra.run.smithi017.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-07-13T06:08:42.312 INFO:teuthology.orchestra.run.smithi014.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-07-13T06:08:42.314 INFO:teuthology.orchestra.run.smithi014.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-07-13T06:08:42.349 INFO:teuthology.orchestra.run.smithi017.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-07-13T06:08:42.351 INFO:teuthology.orchestra.run.smithi017.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-07-13T06:08:42.390 INFO:teuthology.orchestra.run.smithi014.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-07-13T06:08:42.391 INFO:teuthology.orchestra.run.smithi014.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-07-13T06:08:42.397 INFO:teuthology.orchestra.run.smithi014.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-07-13T06:08:42.397 INFO:teuthology.orchestra.run.smithi014.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-07-13T06:08:42.398 INFO:teuthology.orchestra.run.smithi014.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-07-13T06:08:42.401 INFO:teuthology.orchestra.run.smithi014.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-07-13T06:08:42.402 INFO:teuthology.orchestra.run.smithi014.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-07-13T06:08:42.403 INFO:teuthology.orchestra.run.smithi014.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-07-13T06:08:42.421 INFO:teuthology.orchestra.run.smithi017.stdout:Get:62 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-3402-g1ac5c74-1bionic [5,551 kB] 2020-07-13T06:08:42.430 INFO:teuthology.orchestra.run.smithi017.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-07-13T06:08:42.431 INFO:teuthology.orchestra.run.smithi017.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-07-13T06:08:42.436 INFO:teuthology.orchestra.run.smithi017.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-07-13T06:08:42.437 INFO:teuthology.orchestra.run.smithi017.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-07-13T06:08:42.438 INFO:teuthology.orchestra.run.smithi017.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-07-13T06:08:42.441 INFO:teuthology.orchestra.run.smithi017.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-07-13T06:08:42.442 INFO:teuthology.orchestra.run.smithi017.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-07-13T06:08:42.443 INFO:teuthology.orchestra.run.smithi017.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-07-13T06:08:42.468 INFO:teuthology.orchestra.run.smithi014.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-07-13T06:08:42.473 INFO:teuthology.orchestra.run.smithi014.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-07-13T06:08:42.510 INFO:teuthology.orchestra.run.smithi017.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-07-13T06:08:42.514 INFO:teuthology.orchestra.run.smithi017.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-07-13T06:08:42.546 INFO:teuthology.orchestra.run.smithi014.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-07-13T06:08:42.547 INFO:teuthology.orchestra.run.smithi014.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-07-13T06:08:42.548 INFO:teuthology.orchestra.run.smithi014.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-07-13T06:08:42.549 INFO:teuthology.orchestra.run.smithi014.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-07-13T06:08:42.551 INFO:teuthology.orchestra.run.smithi014.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-07-13T06:08:42.555 INFO:teuthology.orchestra.run.smithi014.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-07-13T06:08:42.557 INFO:teuthology.orchestra.run.smithi014.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-07-13T06:08:42.557 INFO:teuthology.orchestra.run.smithi014.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-07-13T06:08:42.591 INFO:teuthology.orchestra.run.smithi017.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-07-13T06:08:42.592 INFO:teuthology.orchestra.run.smithi017.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-07-13T06:08:42.592 INFO:teuthology.orchestra.run.smithi017.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-07-13T06:08:42.593 INFO:teuthology.orchestra.run.smithi017.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-07-13T06:08:42.595 INFO:teuthology.orchestra.run.smithi017.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-07-13T06:08:42.599 INFO:teuthology.orchestra.run.smithi017.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-07-13T06:08:42.600 INFO:teuthology.orchestra.run.smithi017.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-07-13T06:08:42.601 INFO:teuthology.orchestra.run.smithi017.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-07-13T06:08:42.624 INFO:teuthology.orchestra.run.smithi014.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-07-13T06:08:42.627 INFO:teuthology.orchestra.run.smithi014.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-07-13T06:08:42.671 INFO:teuthology.orchestra.run.smithi017.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-3402-g1ac5c74-1bionic [146 kB] 2020-07-13T06:08:42.672 INFO:teuthology.orchestra.run.smithi017.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-07-13T06:08:42.672 INFO:teuthology.orchestra.run.smithi017.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-3402-g1ac5c74-1bionic [1,351 kB] 2020-07-13T06:08:42.674 INFO:teuthology.orchestra.run.smithi017.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-07-13T06:08:42.703 INFO:teuthology.orchestra.run.smithi014.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-07-13T06:08:42.705 INFO:teuthology.orchestra.run.smithi014.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-07-13T06:08:42.706 INFO:teuthology.orchestra.run.smithi014.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-07-13T06:08:42.752 INFO:teuthology.orchestra.run.smithi017.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-07-13T06:08:42.754 INFO:teuthology.orchestra.run.smithi017.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-07-13T06:08:42.756 INFO:teuthology.orchestra.run.smithi017.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-07-13T06:08:42.774 INFO:teuthology.orchestra.run.smithi017.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-3402-g1ac5c74-1bionic [6,575 kB] 2020-07-13T06:08:42.804 INFO:teuthology.orchestra.run.smithi014.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-07-13T06:08:42.805 INFO:teuthology.orchestra.run.smithi014.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-07-13T06:08:42.805 INFO:teuthology.orchestra.run.smithi014.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.2 [329 kB] 2020-07-13T06:08:42.808 INFO:teuthology.orchestra.run.smithi014.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-07-13T06:08:42.808 INFO:teuthology.orchestra.run.smithi014.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-07-13T06:08:42.853 INFO:teuthology.orchestra.run.smithi017.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-07-13T06:08:42.854 INFO:teuthology.orchestra.run.smithi017.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-07-13T06:08:42.854 INFO:teuthology.orchestra.run.smithi017.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.2 [329 kB] 2020-07-13T06:08:42.857 INFO:teuthology.orchestra.run.smithi017.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-07-13T06:08:42.857 INFO:teuthology.orchestra.run.smithi017.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-07-13T06:08:42.857 INFO:teuthology.orchestra.run.smithi017.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-07-13T06:08:42.858 INFO:teuthology.orchestra.run.smithi017.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-07-13T06:08:42.860 INFO:teuthology.orchestra.run.smithi014.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-07-13T06:08:42.861 INFO:teuthology.orchestra.run.smithi014.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-07-13T06:08:42.867 INFO:teuthology.orchestra.run.smithi014.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-07-13T06:08:42.870 INFO:teuthology.orchestra.run.smithi014.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-07-13T06:08:42.915 INFO:teuthology.orchestra.run.smithi017.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-07-13T06:08:42.915 INFO:teuthology.orchestra.run.smithi017.stdout:Get:97 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-07-13T06:08:43.021 INFO:teuthology.orchestra.run.smithi017.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-3402-g1ac5c74-1bionic [24.6 MB] 2020-07-13T06:08:44.034 INFO:teuthology.orchestra.run.smithi017.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-3402-g1ac5c74-1bionic [3,812 B] 2020-07-13T06:08:44.034 INFO:teuthology.orchestra.run.smithi017.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-3402-g1ac5c74-1bionic [541 kB] 2020-07-13T06:08:44.055 INFO:teuthology.orchestra.run.smithi017.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-3402-g1ac5c74-1bionic [1,949 kB] 2020-07-13T06:08:44.128 INFO:teuthology.orchestra.run.smithi017.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-3402-g1ac5c74-1bionic [43.0 kB] 2020-07-13T06:08:44.135 INFO:teuthology.orchestra.run.smithi017.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-3402-g1ac5c74-1bionic [44.4 kB] 2020-07-13T06:08:44.136 INFO:teuthology.orchestra.run.smithi017.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-3402-g1ac5c74-1bionic [3,122 kB] 2020-07-13T06:08:44.256 INFO:teuthology.orchestra.run.smithi017.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 16.0.0-3402-g1ac5c74-1bionic [34.1 kB] 2020-07-13T06:08:44.257 INFO:teuthology.orchestra.run.smithi017.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-3402-g1ac5c74-1bionic [7,887 kB] 2020-07-13T06:08:44.542 INFO:teuthology.orchestra.run.smithi014.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-3402-g1ac5c74-1bionic [5,551 kB] 2020-07-13T06:08:44.575 INFO:teuthology.orchestra.run.smithi017.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-3402-g1ac5c74-1bionic [3,928 B] 2020-07-13T06:08:44.576 INFO:teuthology.orchestra.run.smithi017.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-3402-g1ac5c74-1bionic [22.3 kB] 2020-07-13T06:08:44.576 INFO:teuthology.orchestra.run.smithi017.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-3402-g1ac5c74-1bionic [47.5 MB] 2020-07-13T06:08:45.851 INFO:teuthology.orchestra.run.smithi014.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-3402-g1ac5c74-1bionic [146 kB] 2020-07-13T06:08:45.878 INFO:teuthology.orchestra.run.smithi014.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-3402-g1ac5c74-1bionic [1,351 kB] 2020-07-13T06:08:46.188 INFO:teuthology.orchestra.run.smithi014.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-3402-g1ac5c74-1bionic [6,575 kB] 2020-07-13T06:08:46.390 INFO:teuthology.orchestra.run.smithi017.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-3402-g1ac5c74-1bionic [19.2 kB] 2020-07-13T06:08:46.391 INFO:teuthology.orchestra.run.smithi017.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-3402-g1ac5c74-1bionic [9,243 kB] 2020-07-13T06:08:46.739 INFO:teuthology.orchestra.run.smithi017.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-3402-g1ac5c74-1bionic [76.4 kB] 2020-07-13T06:08:47.183 INFO:teuthology.orchestra.run.smithi014.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-3402-g1ac5c74-1bionic [24.6 MB] 2020-07-13T06:08:47.473 INFO:teuthology.orchestra.run.smithi017.stdout:Fetched 164 MB in 6s (27.3 MB/s) 2020-07-13T06:08:47.620 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-uritemplate. 2020-07-13T06:08:47.659 INFO:teuthology.orchestra.run.smithi017.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102053 files and directories currently installed.) 2020-07-13T06:08:47.663 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-07-13T06:08:47.702 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-07-13T06:08:47.980 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-07-13T06:08:47.992 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-07-13T06:08:48.071 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-07-13T06:08:48.403 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../002-librbd1_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:08:48.525 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking librbd1 (16.0.0-3402-g1ac5c74-1bionic) over (12.2.13-0ubuntu0.18.04.2) ... 2020-07-13T06:08:49.048 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../003-librados2_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:08:49.154 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking librados2 (16.0.0-3402-g1ac5c74-1bionic) over (12.2.13-0ubuntu0.18.04.2) ... 2020-07-13T06:08:49.408 INFO:teuthology.orchestra.run.smithi014.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-3402-g1ac5c74-1bionic [3,812 B] 2020-07-13T06:08:49.408 INFO:teuthology.orchestra.run.smithi014.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-3402-g1ac5c74-1bionic [541 kB] 2020-07-13T06:08:49.449 INFO:teuthology.orchestra.run.smithi014.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-3402-g1ac5c74-1bionic [1,949 kB] 2020-07-13T06:08:49.603 INFO:teuthology.orchestra.run.smithi014.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-3402-g1ac5c74-1bionic [43.0 kB] 2020-07-13T06:08:49.609 INFO:teuthology.orchestra.run.smithi014.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-3402-g1ac5c74-1bionic [44.4 kB] 2020-07-13T06:08:49.611 INFO:teuthology.orchestra.run.smithi014.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-3402-g1ac5c74-1bionic [3,122 kB] 2020-07-13T06:08:49.783 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libcephfs2. 2020-07-13T06:08:49.795 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:08:49.833 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libcephfs2 (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:08:49.853 INFO:teuthology.orchestra.run.smithi014.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 16.0.0-3402-g1ac5c74-1bionic [34.1 kB] 2020-07-13T06:08:49.856 INFO:teuthology.orchestra.run.smithi014.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-3402-g1ac5c74-1bionic [7,887 kB] 2020-07-13T06:08:50.110 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-rados. 2020-07-13T06:08:50.122 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../005-python3-rados_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:08:50.160 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-rados (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:08:50.434 INFO:teuthology.orchestra.run.smithi014.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-3402-g1ac5c74-1bionic [3,928 B] 2020-07-13T06:08:50.435 INFO:teuthology.orchestra.run.smithi014.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-3402-g1ac5c74-1bionic [22.3 kB] 2020-07-13T06:08:50.435 INFO:teuthology.orchestra.run.smithi014.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-3402-g1ac5c74-1bionic [47.5 MB] 2020-07-13T06:08:50.504 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-07-13T06:08:50.516 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-3402-g1ac5c74-1bionic_all.deb ... 2020-07-13T06:08:50.554 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-ceph-argparse (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:08:50.832 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-cephfs. 2020-07-13T06:08:50.843 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:08:50.884 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-cephfs (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:08:51.210 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-ceph-common. 2020-07-13T06:08:51.222 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-3402-g1ac5c74-1bionic_all.deb ... 2020-07-13T06:08:51.259 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-ceph-common (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:08:51.578 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-prettytable. 2020-07-13T06:08:51.591 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-07-13T06:08:51.628 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-07-13T06:08:51.882 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-rbd. 2020-07-13T06:08:51.893 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:08:51.931 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-rbd (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:08:52.243 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-07-13T06:08:52.255 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-07-13T06:08:52.284 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-07-13T06:08:52.611 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-07-13T06:08:52.623 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-07-13T06:08:52.661 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-07-13T06:08:53.005 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package librgw2. 2020-07-13T06:08:53.017 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../013-librgw2_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:08:53.064 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking librgw2 (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:08:53.541 INFO:teuthology.orchestra.run.smithi014.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-3402-g1ac5c74-1bionic [19.2 kB] 2020-07-13T06:08:53.542 INFO:teuthology.orchestra.run.smithi014.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-3402-g1ac5c74-1bionic [9,243 kB] 2020-07-13T06:08:53.616 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-rgw. 2020-07-13T06:08:53.628 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:08:53.666 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-rgw (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:08:53.986 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-07-13T06:08:53.997 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-07-13T06:08:54.035 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-07-13T06:08:54.186 INFO:teuthology.orchestra.run.smithi014.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/master/1ac5c74502da6dc8ad965625df452eb42a846489/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-3402-g1ac5c74-1bionic [76.4 kB] 2020-07-13T06:08:54.288 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-07-13T06:08:54.301 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-07-13T06:08:54.330 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-07-13T06:08:54.691 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package liboath0. 2020-07-13T06:08:54.703 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-07-13T06:08:54.741 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-07-13T06:08:54.919 INFO:teuthology.orchestra.run.smithi014.stdout:Fetched 164 MB in 13s (12.2 MB/s) 2020-07-13T06:08:55.035 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libradosstriper1. 2020-07-13T06:08:55.040 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-uritemplate. 2020-07-13T06:08:55.048 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:08:55.078 INFO:teuthology.orchestra.run.smithi014.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102053 files and directories currently installed.) 2020-07-13T06:08:55.080 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-07-13T06:08:55.085 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libradosstriper1 (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:08:55.114 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-07-13T06:08:55.408 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-07-13T06:08:55.420 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-07-13T06:08:55.521 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-common. 2020-07-13T06:08:55.534 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../019-ceph-common_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:08:55.571 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-common (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:08:55.625 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-07-13T06:08:55.973 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../002-librbd1_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:08:56.102 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking librbd1 (16.0.0-3402-g1ac5c74-1bionic) over (12.2.13-0ubuntu0.18.04.2) ... 2020-07-13T06:08:56.602 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../003-librados2_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:08:56.699 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking librados2 (16.0.0-3402-g1ac5c74-1bionic) over (12.2.13-0ubuntu0.18.04.2) ... 2020-07-13T06:08:57.319 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package libcephfs2. 2020-07-13T06:08:57.334 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:08:57.369 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-base. 2020-07-13T06:08:57.370 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking libcephfs2 (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:08:57.382 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../020-ceph-base_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:08:57.448 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-base (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:08:57.672 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-rados. 2020-07-13T06:08:57.684 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../005-python3-rados_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:08:57.722 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-rados (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:08:58.027 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-07-13T06:08:58.036 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-3402-g1ac5c74-1bionic_all.deb ... 2020-07-13T06:08:58.066 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-ceph-argparse (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:08:58.310 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-cephfs. 2020-07-13T06:08:58.322 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:08:58.323 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-dateutil. 2020-07-13T06:08:58.334 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-07-13T06:08:58.356 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-cephfs (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:08:58.365 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-07-13T06:08:58.640 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-asn1crypto. 2020-07-13T06:08:58.647 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-07-13T06:08:58.685 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-07-13T06:08:58.821 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-ceph-common. 2020-07-13T06:08:58.833 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-3402-g1ac5c74-1bionic_all.deb ... 2020-07-13T06:08:58.871 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-ceph-common (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:08:59.182 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-prettytable. 2020-07-13T06:08:59.193 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-07-13T06:08:59.220 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-cffi-backend. 2020-07-13T06:08:59.232 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-07-13T06:08:59.232 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-07-13T06:08:59.270 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-07-13T06:08:59.476 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-rbd. 2020-07-13T06:08:59.489 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:08:59.526 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-rbd (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:08:59.614 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-cryptography. 2020-07-13T06:08:59.626 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.3_amd64.deb ... 2020-07-13T06:08:59.673 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-07-13T06:08:59.820 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-07-13T06:08:59.832 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-07-13T06:08:59.870 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-07-13T06:09:00.009 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-openssl. 2020-07-13T06:09:00.021 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-07-13T06:09:00.059 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-07-13T06:09:00.206 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-07-13T06:09:00.219 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-07-13T06:09:00.257 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-07-13T06:09:00.287 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-07-13T06:09:00.298 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-3402-g1ac5c74-1bionic_all.deb ... 2020-07-13T06:09:00.328 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-modules-core (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:00.575 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package librgw2. 2020-07-13T06:09:00.588 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../013-librgw2_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:00.625 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking librgw2 (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:00.683 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-bcrypt. 2020-07-13T06:09:00.697 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-07-13T06:09:00.732 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-07-13T06:09:01.027 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-cherrypy3. 2020-07-13T06:09:01.038 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-07-13T06:09:01.077 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-07-13T06:09:01.086 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-rgw. 2020-07-13T06:09:01.092 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:01.127 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-rgw (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:01.405 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-jwt. 2020-07-13T06:09:01.416 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-07-13T06:09:01.464 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-07-13T06:09:01.474 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-07-13T06:09:01.481 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-07-13T06:09:01.514 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-07-13T06:09:01.766 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-logutils. 2020-07-13T06:09:01.778 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-07-13T06:09:01.799 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-07-13T06:09:01.808 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-07-13T06:09:01.812 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-07-13T06:09:01.849 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-07-13T06:09:02.094 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-markupsafe. 2020-07-13T06:09:02.107 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-07-13T06:09:02.144 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-07-13T06:09:02.185 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package liboath0. 2020-07-13T06:09:02.198 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-07-13T06:09:02.235 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-07-13T06:09:02.446 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-mako. 2020-07-13T06:09:02.456 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-07-13T06:09:02.496 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-07-13T06:09:02.505 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package libradosstriper1. 2020-07-13T06:09:02.515 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:02.546 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking libradosstriper1 (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:02.824 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-simplegeneric. 2020-07-13T06:09:02.835 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-07-13T06:09:02.874 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-07-13T06:09:03.016 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package ceph-common. 2020-07-13T06:09:03.028 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../019-ceph-common_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:03.066 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking ceph-common (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:03.152 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-singledispatch. 2020-07-13T06:09:03.164 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-07-13T06:09:03.202 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-07-13T06:09:03.512 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-webob. 2020-07-13T06:09:03.524 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-07-13T06:09:03.562 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-07-13T06:09:03.882 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-bs4. 2020-07-13T06:09:03.894 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-07-13T06:09:03.932 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-07-13T06:09:04.376 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-waitress. 2020-07-13T06:09:04.387 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-07-13T06:09:04.426 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-07-13T06:09:04.746 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-tempita. 2020-07-13T06:09:04.757 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-07-13T06:09:04.795 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-07-13T06:09:04.797 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package ceph-base. 2020-07-13T06:09:04.810 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../020-ceph-base_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:04.847 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking ceph-base (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:05.098 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-paste. 2020-07-13T06:09:05.110 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-07-13T06:09:05.148 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-07-13T06:09:05.468 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-07-13T06:09:05.479 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-07-13T06:09:05.534 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-07-13T06:09:05.785 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-dateutil. 2020-07-13T06:09:05.797 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-07-13T06:09:05.835 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-07-13T06:09:05.862 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pastedeploy. 2020-07-13T06:09:05.873 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-07-13T06:09:05.912 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-07-13T06:09:06.146 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-asn1crypto. 2020-07-13T06:09:06.158 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-07-13T06:09:06.196 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-07-13T06:09:06.198 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-webtest. 2020-07-13T06:09:06.210 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-07-13T06:09:06.239 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-07-13T06:09:06.457 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-cffi-backend. 2020-07-13T06:09:06.468 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-07-13T06:09:06.507 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-07-13T06:09:06.559 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pecan. 2020-07-13T06:09:06.571 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-07-13T06:09:06.609 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-07-13T06:09:06.817 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-cryptography. 2020-07-13T06:09:06.828 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.3_amd64.deb ... 2020-07-13T06:09:06.867 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-07-13T06:09:06.930 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-werkzeug. 2020-07-13T06:09:06.943 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-07-13T06:09:06.980 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-07-13T06:09:07.229 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-openssl. 2020-07-13T06:09:07.241 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-07-13T06:09:07.279 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-07-13T06:09:07.317 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr. 2020-07-13T06:09:07.328 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:07.366 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:07.548 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-07-13T06:09:07.559 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-3402-g1ac5c74-1bionic_all.deb ... 2020-07-13T06:09:07.598 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking ceph-mgr-modules-core (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:07.820 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mon. 2020-07-13T06:09:07.829 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:07.869 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mon (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:07.970 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-bcrypt. 2020-07-13T06:09:07.980 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-07-13T06:09:08.019 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-07-13T06:09:08.321 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-cherrypy3. 2020-07-13T06:09:08.333 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-07-13T06:09:08.371 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-07-13T06:09:08.808 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-jwt. 2020-07-13T06:09:08.814 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-osd. 2020-07-13T06:09:08.820 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-07-13T06:09:08.826 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:08.863 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-07-13T06:09:08.864 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-osd (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:09.127 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-logutils. 2020-07-13T06:09:09.139 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-07-13T06:09:09.177 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-07-13T06:09:09.455 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-markupsafe. 2020-07-13T06:09:09.468 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-07-13T06:09:09.505 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-07-13T06:09:09.808 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-mako. 2020-07-13T06:09:09.820 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-07-13T06:09:09.858 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-07-13T06:09:10.169 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-simplegeneric. 2020-07-13T06:09:10.180 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-07-13T06:09:10.219 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-07-13T06:09:10.496 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-singledispatch. 2020-07-13T06:09:10.509 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-07-13T06:09:10.546 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-07-13T06:09:10.824 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-webob. 2020-07-13T06:09:10.836 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-07-13T06:09:10.861 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph. 2020-07-13T06:09:10.874 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../048-ceph_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:10.874 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-07-13T06:09:10.911 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:11.151 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-bs4. 2020-07-13T06:09:11.165 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-07-13T06:09:11.226 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-07-13T06:09:11.230 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-fuse. 2020-07-13T06:09:11.244 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:11.280 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-fuse (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:11.529 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-waitress. 2020-07-13T06:09:11.542 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-07-13T06:09:11.579 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-07-13T06:09:11.633 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mds. 2020-07-13T06:09:11.645 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:11.675 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mds (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:11.899 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-tempita. 2020-07-13T06:09:11.912 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-07-13T06:09:11.948 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-07-13T06:09:12.194 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package cephadm. 2020-07-13T06:09:12.206 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../051-cephadm_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:12.244 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking cephadm (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:12.251 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-paste. 2020-07-13T06:09:12.263 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-07-13T06:09:12.301 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-07-13T06:09:12.539 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-jinja2. 2020-07-13T06:09:12.551 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-07-13T06:09:12.589 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-07-13T06:09:12.621 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-07-13T06:09:12.633 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-07-13T06:09:12.671 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-07-13T06:09:12.917 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-07-13T06:09:12.929 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-3402-g1ac5c74-1bionic_all.deb ... 2020-07-13T06:09:12.957 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-pastedeploy. 2020-07-13T06:09:12.966 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-cephadm (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:12.969 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-07-13T06:09:13.023 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-07-13T06:09:13.326 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-webtest. 2020-07-13T06:09:13.338 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-07-13T06:09:13.376 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-07-13T06:09:13.411 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-repoze.lru. 2020-07-13T06:09:13.424 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-07-13T06:09:13.461 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-07-13T06:09:13.695 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-pecan. 2020-07-13T06:09:13.708 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-07-13T06:09:13.764 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-routes. 2020-07-13T06:09:13.775 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-07-13T06:09:13.805 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-07-13T06:09:14.125 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-07-13T06:09:14.136 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-3402-g1ac5c74-1bionic_all.deb ... 2020-07-13T06:09:14.176 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-dashboard (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:14.403 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-07-13T06:09:14.760 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-werkzeug. 2020-07-13T06:09:14.771 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-07-13T06:09:14.810 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-07-13T06:09:15.163 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package ceph-mgr. 2020-07-13T06:09:15.175 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:15.213 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking ceph-mgr (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:15.582 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2020-07-13T06:09:15.594 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../057-ceph-mgr-diskprediction-cloud_16.0.0-3402-g1ac5c74-1bionic_all.deb ... 2020-07-13T06:09:15.631 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-diskprediction-cloud (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:15.684 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package ceph-mon. 2020-07-13T06:09:15.695 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:15.733 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking ceph-mon (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:15.926 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-numpy. 2020-07-13T06:09:15.938 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-07-13T06:09:15.976 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-07-13T06:09:16.590 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-decorator. 2020-07-13T06:09:16.605 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2020-07-13T06:09:16.636 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package ceph-osd. 2020-07-13T06:09:16.639 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-07-13T06:09:16.649 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:16.686 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking ceph-osd (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:16.909 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-scipy. 2020-07-13T06:09:16.921 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-07-13T06:09:16.951 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-07-13T06:09:18.185 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-07-13T06:09:18.198 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-07-13T06:09:18.235 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-07-13T06:09:18.692 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package ceph. 2020-07-13T06:09:18.697 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-joblib. 2020-07-13T06:09:18.704 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../048-ceph_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:18.708 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2020-07-13T06:09:18.742 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking ceph (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:18.747 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-joblib (0.11-1) ... 2020-07-13T06:09:19.002 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package ceph-fuse. 2020-07-13T06:09:19.013 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:19.044 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking ceph-fuse (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:19.076 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-sklearn. 2020-07-13T06:09:19.091 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2020-07-13T06:09:19.125 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-07-13T06:09:19.431 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package ceph-mds. 2020-07-13T06:09:19.442 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:19.489 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking ceph-mds (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:19.742 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-07-13T06:09:19.754 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_16.0.0-3402-g1ac5c74-1bionic_all.deb ... 2020-07-13T06:09:19.791 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:19.984 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package cephadm. 2020-07-13T06:09:19.996 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../051-cephadm_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:20.034 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking cephadm (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:20.270 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-jinja2. 2020-07-13T06:09:20.286 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-07-13T06:09:20.320 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-07-13T06:09:20.631 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-07-13T06:09:20.644 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-3402-g1ac5c74-1bionic_all.deb ... 2020-07-13T06:09:20.681 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking ceph-mgr-cephadm (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:20.879 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-monotonic. 2020-07-13T06:09:20.890 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2020-07-13T06:09:20.930 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-07-13T06:09:21.018 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-repoze.lru. 2020-07-13T06:09:21.030 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-07-13T06:09:21.068 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-07-13T06:09:21.224 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-fasteners. 2020-07-13T06:09:21.235 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2020-07-13T06:09:21.274 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-07-13T06:09:21.354 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-routes. 2020-07-13T06:09:21.366 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-07-13T06:09:21.404 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-07-13T06:09:21.585 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-httplib2. 2020-07-13T06:09:21.598 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-07-13T06:09:21.635 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-07-13T06:09:21.707 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-07-13T06:09:21.719 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-3402-g1ac5c74-1bionic_all.deb ... 2020-07-13T06:09:21.757 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking ceph-mgr-dashboard (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:21.913 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pyasn1. 2020-07-13T06:09:21.927 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2020-07-13T06:09:21.963 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-07-13T06:09:22.274 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-07-13T06:09:22.286 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-07-13T06:09:22.324 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-07-13T06:09:22.614 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2020-07-13T06:09:22.627 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-rsa. 2020-07-13T06:09:22.628 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../057-ceph-mgr-diskprediction-cloud_16.0.0-3402-g1ac5c74-1bionic_all.deb ... 2020-07-13T06:09:22.641 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2020-07-13T06:09:22.664 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking ceph-mgr-diskprediction-cloud (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:22.677 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-07-13T06:09:22.959 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-numpy. 2020-07-13T06:09:22.975 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-07-13T06:09:22.976 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-07-13T06:09:22.986 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-07-13T06:09:23.017 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-07-13T06:09:23.021 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-07-13T06:09:23.442 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-oauth2client. 2020-07-13T06:09:23.453 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2020-07-13T06:09:23.484 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-07-13T06:09:23.597 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-decorator. 2020-07-13T06:09:23.610 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2020-07-13T06:09:23.639 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-07-13T06:09:23.810 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-setuptools. 2020-07-13T06:09:23.817 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2020-07-13T06:09:23.870 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-07-13T06:09:23.942 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-scipy. 2020-07-13T06:09:23.955 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-07-13T06:09:23.992 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-07-13T06:09:24.240 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-websocket. 2020-07-13T06:09:24.253 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-07-13T06:09:24.290 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-07-13T06:09:24.609 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-kubernetes. 2020-07-13T06:09:24.622 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-07-13T06:09:24.659 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-07-13T06:09:25.028 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-07-13T06:09:25.039 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_16.0.0-3402-g1ac5c74-1bionic_all.deb ... 2020-07-13T06:09:25.080 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:25.225 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-07-13T06:09:25.236 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-07-13T06:09:25.268 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-07-13T06:09:25.401 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-json-pointer. 2020-07-13T06:09:25.414 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2020-07-13T06:09:25.451 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-07-13T06:09:25.678 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-joblib. 2020-07-13T06:09:25.691 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2020-07-13T06:09:25.728 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-joblib (0.11-1) ... 2020-07-13T06:09:25.787 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-jsonpatch. 2020-07-13T06:09:25.801 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-07-13T06:09:25.840 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-07-13T06:09:26.073 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-sklearn. 2020-07-13T06:09:26.085 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2020-07-13T06:09:26.131 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-07-13T06:09:26.149 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-07-13T06:09:26.160 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../079-ceph-mgr-rook_16.0.0-3402-g1ac5c74-1bionic_all.deb ... 2020-07-13T06:09:26.199 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-mgr-rook (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:26.527 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libonig4:amd64. 2020-07-13T06:09:26.540 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2020-07-13T06:09:26.598 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-07-13T06:09:26.611 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_16.0.0-3402-g1ac5c74-1bionic_all.deb ... 2020-07-13T06:09:26.639 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:26.662 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-07-13T06:09:26.965 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libjq1:amd64. 2020-07-13T06:09:26.976 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2020-07-13T06:09:27.015 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-07-13T06:09:27.284 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package jq. 2020-07-13T06:09:27.298 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2020-07-13T06:09:27.343 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-07-13T06:09:27.622 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package socat. 2020-07-13T06:09:27.636 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-07-13T06:09:27.681 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-07-13T06:09:27.753 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-monotonic. 2020-07-13T06:09:27.764 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2020-07-13T06:09:27.803 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-07-13T06:09:28.026 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-07-13T06:09:28.038 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-07-13T06:09:28.076 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-07-13T06:09:28.106 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-fasteners. 2020-07-13T06:09:28.118 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2020-07-13T06:09:28.156 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-07-13T06:09:28.355 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package xmlstarlet. 2020-07-13T06:09:28.369 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2020-07-13T06:09:28.413 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-07-13T06:09:28.442 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-httplib2. 2020-07-13T06:09:28.454 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-07-13T06:09:28.492 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-07-13T06:09:28.736 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ceph-test. 2020-07-13T06:09:28.747 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../086-ceph-test_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:28.792 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ceph-test (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:28.794 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-pyasn1. 2020-07-13T06:09:28.807 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2020-07-13T06:09:28.845 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-07-13T06:09:29.398 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-07-13T06:09:29.411 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-07-13T06:09:29.448 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-07-13T06:09:29.768 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-rsa. 2020-07-13T06:09:29.780 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2020-07-13T06:09:29.818 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-07-13T06:09:30.104 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-07-13T06:09:30.117 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-07-13T06:09:30.154 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-07-13T06:09:30.567 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-oauth2client. 2020-07-13T06:09:30.581 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2020-07-13T06:09:30.625 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-07-13T06:09:30.961 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-setuptools. 2020-07-13T06:09:30.973 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2020-07-13T06:09:31.021 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-07-13T06:09:31.423 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-websocket. 2020-07-13T06:09:31.438 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-07-13T06:09:31.481 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-07-13T06:09:31.801 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-kubernetes. 2020-07-13T06:09:31.813 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-07-13T06:09:31.851 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-07-13T06:09:32.222 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-07-13T06:09:32.234 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_16.0.0-3402-g1ac5c74-1bionic_all.deb ... 2020-07-13T06:09:32.272 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:32.329 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libcephfs-dev. 2020-07-13T06:09:32.340 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../087-libcephfs-dev_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:32.378 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libcephfs-dev (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:32.602 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-json-pointer. 2020-07-13T06:09:32.614 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2020-07-13T06:09:32.651 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-07-13T06:09:32.694 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python-attr. 2020-07-13T06:09:32.707 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-07-13T06:09:32.744 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python-attr (17.4.0-2) ... 2020-07-13T06:09:32.988 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-jsonpatch. 2020-07-13T06:09:33.001 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-07-13T06:09:33.033 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-07-13T06:09:33.047 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python-funcsigs. 2020-07-13T06:09:33.060 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-07-13T06:09:33.099 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-07-13T06:09:33.324 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-07-13T06:09:33.336 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../079-ceph-mgr-rook_16.0.0-3402-g1ac5c74-1bionic_all.deb ... 2020-07-13T06:09:33.374 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking ceph-mgr-rook (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:33.392 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python-pluggy. 2020-07-13T06:09:33.404 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-07-13T06:09:33.442 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-07-13T06:09:33.703 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package libonig4:amd64. 2020-07-13T06:09:33.714 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2020-07-13T06:09:33.727 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python-py. 2020-07-13T06:09:33.741 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-07-13T06:09:33.794 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python-py (1.5.2-1) ... 2020-07-13T06:09:33.844 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-07-13T06:09:34.139 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python-pytest. 2020-07-13T06:09:34.152 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-07-13T06:09:34.157 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package libjq1:amd64. 2020-07-13T06:09:34.170 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2020-07-13T06:09:34.189 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-07-13T06:09:34.207 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-07-13T06:09:34.519 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package jq. 2020-07-13T06:09:34.531 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2020-07-13T06:09:34.534 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-attr. 2020-07-13T06:09:34.547 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-07-13T06:09:34.568 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-07-13T06:09:34.584 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-07-13T06:09:34.807 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package socat. 2020-07-13T06:09:34.821 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-07-13T06:09:34.865 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-07-13T06:09:34.899 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-cycler. 2020-07-13T06:09:34.908 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-07-13T06:09:34.945 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-07-13T06:09:35.246 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-07-13T06:09:35.247 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-webencodings. 2020-07-13T06:09:35.258 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-07-13T06:09:35.261 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-07-13T06:09:35.296 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-07-13T06:09:35.306 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-07-13T06:09:35.575 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package xmlstarlet. 2020-07-13T06:09:35.588 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2020-07-13T06:09:35.592 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-html5lib. 2020-07-13T06:09:35.604 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-07-13T06:09:35.625 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-07-13T06:09:35.642 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-07-13T06:09:35.970 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package ceph-test. 2020-07-13T06:09:35.971 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-07-13T06:09:35.980 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../086-ceph-test_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:35.984 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-07-13T06:09:36.020 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking ceph-test (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:36.039 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-07-13T06:09:36.468 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pyparsing. 2020-07-13T06:09:36.482 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-07-13T06:09:36.527 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-07-13T06:09:36.859 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-tz. 2020-07-13T06:09:36.872 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-07-13T06:09:36.912 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-tz (2018.3-2) ... 2020-07-13T06:09:37.212 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-matplotlib. 2020-07-13T06:09:37.224 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-07-13T06:09:37.262 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-07-13T06:09:37.913 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-olefile. 2020-07-13T06:09:37.925 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-07-13T06:09:37.964 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-07-13T06:09:38.319 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pastescript. 2020-07-13T06:09:38.331 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-07-13T06:09:38.369 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-07-13T06:09:38.689 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pil:amd64. 2020-07-13T06:09:38.703 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.2_amd64.deb ... 2020-07-13T06:09:38.747 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.2) ... 2020-07-13T06:09:39.109 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pluggy. 2020-07-13T06:09:39.121 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-07-13T06:09:39.161 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-07-13T06:09:39.436 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-py. 2020-07-13T06:09:39.448 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-07-13T06:09:39.478 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-py (1.5.2-1) ... 2020-07-13T06:09:39.574 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package libcephfs-dev. 2020-07-13T06:09:39.584 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../087-libcephfs-dev_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:39.623 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking libcephfs-dev (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:39.789 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pyinotify. 2020-07-13T06:09:39.801 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-07-13T06:09:39.839 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-07-13T06:09:39.898 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python-attr. 2020-07-13T06:09:39.912 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-07-13T06:09:39.947 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python-attr (17.4.0-2) ... 2020-07-13T06:09:40.159 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-pytest. 2020-07-13T06:09:40.170 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-07-13T06:09:40.208 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-07-13T06:09:40.250 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python-funcsigs. 2020-07-13T06:09:40.263 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-07-13T06:09:40.300 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-07-13T06:09:40.562 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-simplejson. 2020-07-13T06:09:40.575 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-07-13T06:09:40.595 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python-pluggy. 2020-07-13T06:09:40.609 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-07-13T06:09:40.612 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-07-13T06:09:40.653 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-07-13T06:09:40.932 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package python3-tk:amd64. 2020-07-13T06:09:40.943 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-07-13T06:09:40.955 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python-py. 2020-07-13T06:09:40.968 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-07-13T06:09:40.982 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-07-13T06:09:41.005 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python-py (1.5.2-1) ... 2020-07-13T06:09:41.342 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python-pytest. 2020-07-13T06:09:41.362 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-07-13T06:09:41.393 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-07-13T06:09:41.410 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package radosgw. 2020-07-13T06:09:41.424 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../110-radosgw_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:41.468 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking radosgw (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:41.745 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-attr. 2020-07-13T06:09:41.757 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-07-13T06:09:41.795 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-07-13T06:09:42.056 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-cycler. 2020-07-13T06:09:42.068 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-07-13T06:09:42.106 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-07-13T06:09:42.400 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-webencodings. 2020-07-13T06:09:42.413 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-07-13T06:09:42.421 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package rbd-fuse. 2020-07-13T06:09:42.434 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:42.452 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-07-13T06:09:42.471 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking rbd-fuse (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:42.761 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-html5lib. 2020-07-13T06:09:42.765 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-07-13T06:09:42.776 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-07-13T06:09:42.811 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-07-13T06:09:43.068 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-07-13T06:09:43.159 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-07-13T06:09:43.169 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-07-13T06:09:43.208 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-07-13T06:09:43.436 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-07-13T06:09:43.545 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-07-13T06:09:43.620 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-pyparsing. 2020-07-13T06:09:43.629 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-07-13T06:09:43.674 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-07-13T06:09:43.993 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-tz. 2020-07-13T06:09:44.005 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-07-13T06:09:44.043 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-tz (2018.3-2) ... 2020-07-13T06:09:44.165 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.2) ... 2020-07-13T06:09:44.348 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-matplotlib. 2020-07-13T06:09:44.358 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-07-13T06:09:44.396 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-07-13T06:09:44.558 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-07-13T06:09:44.844 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-webencodings (0.5-2) ... 2020-07-13T06:09:45.083 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-olefile. 2020-07-13T06:09:45.097 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-07-13T06:09:45.112 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python-attr (17.4.0-2) ... 2020-07-13T06:09:45.133 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-07-13T06:09:45.313 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-07-13T06:09:45.439 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-py (1.5.2-1) ... 2020-07-13T06:09:45.444 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-pastescript. 2020-07-13T06:09:45.456 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-07-13T06:09:45.494 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-07-13T06:09:45.774 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-07-13T06:09:45.866 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-pil:amd64. 2020-07-13T06:09:45.876 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.2_amd64.deb ... 2020-07-13T06:09:45.914 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.2) ... 2020-07-13T06:09:46.098 INFO:teuthology.orchestra.run.smithi017.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-07-13T06:09:46.276 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-pluggy. 2020-07-13T06:09:46.290 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-07-13T06:09:46.326 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-07-13T06:09:46.410 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-07-13T06:09:46.637 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-py. 2020-07-13T06:09:46.650 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-07-13T06:09:46.687 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-ceph-argparse (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:46.695 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-py (1.5.2-1) ... 2020-07-13T06:09:46.979 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-07-13T06:09:47.006 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-pyinotify. 2020-07-13T06:09:47.021 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-07-13T06:09:47.073 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-07-13T06:09:47.281 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-07-13T06:09:47.309 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-pytest. 2020-07-13T06:09:47.322 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-07-13T06:09:47.350 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-07-13T06:09:47.407 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-07-13T06:09:47.662 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-simplejson. 2020-07-13T06:09:47.676 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-07-13T06:09:47.712 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-07-13T06:09:48.023 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package python3-tk:amd64. 2020-07-13T06:09:48.036 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-07-13T06:09:48.073 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-07-13T06:09:48.385 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-07-13T06:09:48.510 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package radosgw. 2020-07-13T06:09:48.524 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../110-radosgw_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:48.568 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking radosgw (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:48.703 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-07-13T06:09:48.978 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python-py (1.5.2-1) ... 2020-07-13T06:09:49.188 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-07-13T06:09:49.523 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-07-13T06:09:49.555 INFO:teuthology.orchestra.run.smithi014.stdout:Selecting previously unselected package rbd-fuse. 2020-07-13T06:09:49.567 INFO:teuthology.orchestra.run.smithi014.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-3402-g1ac5c74-1bionic_amd64.deb ... 2020-07-13T06:09:49.612 INFO:teuthology.orchestra.run.smithi017.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-07-13T06:09:49.680 INFO:teuthology.orchestra.run.smithi014.stdout:Unpacking rbd-fuse (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:49.857 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-07-13T06:09:49.965 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-07-13T06:09:49.983 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-07-13T06:09:50.254 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-07-13T06:09:50.260 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-07-13T06:09:50.536 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-07-13T06:09:50.628 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-07-13T06:09:50.754 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-07-13T06:09:50.829 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-07-13T06:09:51.113 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-07-13T06:09:51.373 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.2) ... 2020-07-13T06:09:51.473 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-07-13T06:09:51.759 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-07-13T06:09:51.816 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-07-13T06:09:52.052 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-webencodings (0.5-2) ... 2020-07-13T06:09:52.117 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up cephadm (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:52.345 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python-attr (17.4.0-2) ... 2020-07-13T06:09:52.550 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-07-13T06:09:52.672 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-py (1.5.2-1) ... 2020-07-13T06:09:52.809 INFO:teuthology.orchestra.run.smithi017.stdout:Adding system user cephadm....done 2020-07-13T06:09:52.982 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-07-13T06:09:53.010 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-07-13T06:09:53.272 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-07-13T06:09:53.337 INFO:teuthology.orchestra.run.smithi014.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-07-13T06:09:53.393 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-07-13T06:09:53.651 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-07-13T06:09:53.720 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-07-13T06:09:53.978 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-ceph-argparse (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:54.012 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-07-13T06:09:54.113 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-07-13T06:09:54.280 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-07-13T06:09:54.450 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-ceph-common (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:54.580 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-07-13T06:09:54.706 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-07-13T06:09:54.752 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-monotonic (1.1-2) ... 2020-07-13T06:09:55.042 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-07-13T06:09:55.169 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-07-13T06:09:55.519 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-07-13T06:09:55.635 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-07-13T06:09:55.887 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-07-13T06:09:55.943 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-07-13T06:09:55.960 INFO:teuthology.orchestra.run.smithi017.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-07-13T06:09:56.001 INFO:teuthology.orchestra.run.smithi017.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-07-13T06:09:56.236 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python-py (1.5.2-1) ... 2020-07-13T06:09:56.248 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-07-13T06:09:56.462 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-07-13T06:09:56.599 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-07-13T06:09:56.797 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-07-13T06:09:56.886 INFO:teuthology.orchestra.run.smithi014.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-07-13T06:09:56.900 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-07-13T06:09:57.130 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-07-13T06:09:57.241 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-07-13T06:09:57.242 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-attr (17.4.0-2) ... 2020-07-13T06:09:57.442 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-07-13T06:09:57.544 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-07-13T06:09:57.743 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-07-13T06:09:57.745 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-07-13T06:09:58.095 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-07-13T06:09:58.138 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-07-13T06:09:58.224 INFO:teuthology.orchestra.run.smithi017.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-07-13T06:09:58.404 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-07-13T06:09:58.465 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up liboath0 (2.6.1-1) ... 2020-07-13T06:09:58.644 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-07-13T06:09:58.767 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-07-13T06:09:58.780 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-07-13T06:09:59.018 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-07-13T06:09:59.126 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-07-13T06:09:59.144 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-07-13T06:09:59.399 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up cephadm (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:09:59.546 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-07-13T06:09:59.647 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-07-13T06:09:59.931 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-tz (2018.3-2) ... 2020-07-13T06:10:00.232 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-07-13T06:10:00.402 INFO:teuthology.orchestra.run.smithi014.stdout:Adding system user cephadm....done 2020-07-13T06:10:00.509 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-07-13T06:10:00.581 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-07-13T06:10:00.810 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-joblib (0.11-1) ... 2020-07-13T06:10:00.874 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-07-13T06:10:01.000 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-07-13T06:10:01.187 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-routes (2.4.1-1) ... 2020-07-13T06:10:01.351 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-07-13T06:10:01.504 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-07-13T06:10:01.661 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-07-13T06:10:01.787 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-07-13T06:10:01.813 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-07-13T06:10:02.130 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-ceph-common (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:02.132 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-07-13T06:10:02.448 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-monotonic (1.1-2) ... 2020-07-13T06:10:02.500 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-07-13T06:10:02.732 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-07-13T06:10:02.793 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up jq (1.5+dfsg-2) ... 2020-07-13T06:10:02.850 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-07-13T06:10:02.919 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-07-13T06:10:03.193 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-07-13T06:10:03.536 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-07-13T06:10:03.626 INFO:teuthology.orchestra.run.smithi014.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-07-13T06:10:03.666 INFO:teuthology.orchestra.run.smithi014.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-07-13T06:10:03.905 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-07-13T06:10:04.124 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-07-13T06:10:04.214 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-07-13T06:10:04.450 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-07-13T06:10:04.515 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-07-13T06:10:04.776 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-07-13T06:10:04.849 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-attr (17.4.0-2) ... 2020-07-13T06:10:04.903 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-07-13T06:10:05.159 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-07-13T06:10:05.280 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python-pytest (3.3.2-2) ... 2020-07-13T06:10:05.360 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-07-13T06:10:05.740 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-07-13T06:10:05.787 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-07-13T06:10:05.856 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-07-13T06:10:05.876 INFO:teuthology.orchestra.run.smithi014.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-07-13T06:10:05.982 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up librados2 (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:06.108 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-07-13T06:10:06.138 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up liboath0 (2.6.1-1) ... 2020-07-13T06:10:06.314 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-07-13T06:10:06.432 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-07-13T06:10:06.434 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libcephfs2 (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:06.560 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libcephfs-dev (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:06.667 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-07-13T06:10:06.687 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-rados (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:06.801 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-07-13T06:10:06.812 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-07-13T06:10:07.113 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-07-13T06:10:07.190 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-07-13T06:10:07.312 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-07-13T06:10:07.415 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-07-13T06:10:07.605 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-tz (2018.3-2) ... 2020-07-13T06:10:07.743 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-07-13T06:10:07.906 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-07-13T06:10:08.182 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-07-13T06:10:08.475 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-joblib (0.11-1) ... 2020-07-13T06:10:08.610 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-07-13T06:10:08.843 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-routes (2.4.1-1) ... 2020-07-13T06:10:09.169 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-07-13T06:10:09.472 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-07-13T06:10:09.542 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-07-13T06:10:09.806 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-07-13T06:10:09.843 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libradosstriper1 (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:09.969 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-cephfs (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:10.173 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-07-13T06:10:10.274 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up librgw2 (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:10.396 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-fuse (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:10.458 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up jq (1.5+dfsg-2) ... 2020-07-13T06:10:10.567 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-07-13T06:10:10.634 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-07-13T06:10:10.634 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-07-13T06:10:10.897 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up librbd1 (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:11.040 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-07-13T06:10:11.130 INFO:teuthology.orchestra.run.smithi017.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-07-13T06:10:11.162 INFO:teuthology.orchestra.run.smithi017.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-07-13T06:10:11.460 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-07-13T06:10:11.755 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-07-13T06:10:12.090 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-07-13T06:10:12.137 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-modules-core (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:12.263 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-07-13T06:10:12.400 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-07-13T06:10:12.517 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-07-13T06:10:12.573 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-rgw (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:12.699 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up rbd-fuse (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:12.825 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up python3-rbd (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:12.894 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python-pytest (3.3.2-2) ... 2020-07-13T06:10:12.951 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-common (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:13.161 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-07-13T06:10:13.287 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-07-13T06:10:13.414 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up librados2 (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:13.468 INFO:teuthology.orchestra.run.smithi017.stdout:Adding group ceph....done 2020-07-13T06:10:13.581 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-07-13T06:10:13.846 INFO:teuthology.orchestra.run.smithi017.stdout:Adding system user ceph....done 2020-07-13T06:10:13.899 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up libcephfs2 (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:14.008 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up libcephfs-dev (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:14.047 INFO:teuthology.orchestra.run.smithi017.stdout:Setting system user ceph properties....done 2020-07-13T06:10:14.126 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-rados (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:14.206 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-07-13T06:10:14.252 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-07-13T06:10:14.431 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-07-13T06:10:14.578 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-07-13T06:10:14.736 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up radosgw (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:14.887 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-07-13T06:10:15.002 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-07-13T06:10:15.003 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-07-13T06:10:15.222 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-07-13T06:10:15.372 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-test (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:15.507 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-base (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:15.845 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-07-13T06:10:16.075 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-07-13T06:10:16.101 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:16.300 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-07-13T06:10:16.301 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-07-13T06:10:16.560 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:16.695 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-osd (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:16.981 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-07-13T06:10:17.126 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-07-13T06:10:17.126 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-07-13T06:10:17.299 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up libradosstriper1 (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:17.372 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-cephadm (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:17.408 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-cephfs (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:17.515 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-diskprediction-cloud (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:17.657 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-rook (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:17.684 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up librgw2 (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:17.791 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mds (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:17.835 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up ceph-fuse (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:18.004 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-07-13T06:10:18.004 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-07-13T06:10:18.062 INFO:teuthology.orchestra.run.smithi014.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-07-13T06:10:18.063 INFO:teuthology.orchestra.run.smithi014.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-07-13T06:10:18.268 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mon (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:18.353 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up librbd1 (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:18.476 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-07-13T06:10:18.476 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-07-13T06:10:18.479 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-07-13T06:10:18.586 INFO:teuthology.orchestra.run.smithi014.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-07-13T06:10:18.618 INFO:teuthology.orchestra.run.smithi014.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-07-13T06:10:18.752 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-k8sevents (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:18.878 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph-mgr-dashboard (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:18.924 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-07-13T06:10:19.021 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ceph (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:19.148 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for systemd (237-3ubuntu10.41) ... 2020-07-13T06:10:19.340 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-07-13T06:10:19.577 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up ceph-mgr-modules-core (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:19.711 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-07-13T06:10:20.020 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-rgw (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:20.146 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up rbd-fuse (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:20.272 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up python3-rbd (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:20.415 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up ceph-common (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:20.601 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-07-13T06:10:20.635 INFO:teuthology.orchestra.run.smithi017.stdout:ureadahead will be reprofiled on next reboot 2020-07-13T06:10:20.690 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.2) ... 2020-07-13T06:10:20.914 INFO:teuthology.orchestra.run.smithi014.stdout:Adding group ceph....done 2020-07-13T06:10:21.205 INFO:teuthology.orchestra.run.smithi017.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-07-13T06:10:21.210 DEBUG:teuthology.parallel:result is None 2020-07-13T06:10:21.329 INFO:teuthology.orchestra.run.smithi014.stdout:Adding system user ceph....done 2020-07-13T06:10:21.529 INFO:teuthology.orchestra.run.smithi014.stdout:Setting system user ceph properties....done 2020-07-13T06:10:21.686 INFO:teuthology.orchestra.run.smithi014.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-07-13T06:10:21.928 INFO:teuthology.orchestra.run.smithi014.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-07-13T06:10:22.211 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up radosgw (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:22.451 INFO:teuthology.orchestra.run.smithi014.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-07-13T06:10:22.451 INFO:teuthology.orchestra.run.smithi014.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-07-13T06:10:22.830 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up ceph-test (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:22.931 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up ceph-base (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:23.270 INFO:teuthology.orchestra.run.smithi014.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-07-13T06:10:23.574 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up ceph-mgr (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:23.802 INFO:teuthology.orchestra.run.smithi014.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-07-13T06:10:23.803 INFO:teuthology.orchestra.run.smithi014.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-07-13T06:10:24.058 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:24.201 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up ceph-osd (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:24.646 INFO:teuthology.orchestra.run.smithi014.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-07-13T06:10:24.646 INFO:teuthology.orchestra.run.smithi014.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-07-13T06:10:24.928 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up ceph-mgr-cephadm (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:25.054 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up ceph-mgr-diskprediction-cloud (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:25.172 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up ceph-mgr-rook (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:25.306 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up ceph-mds (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:25.534 INFO:teuthology.orchestra.run.smithi014.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-07-13T06:10:25.535 INFO:teuthology.orchestra.run.smithi014.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-07-13T06:10:25.800 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up ceph-mon (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:26.019 INFO:teuthology.orchestra.run.smithi014.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-07-13T06:10:26.020 INFO:teuthology.orchestra.run.smithi014.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-07-13T06:10:26.301 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up ceph-mgr-k8sevents (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:26.427 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up ceph-mgr-dashboard (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:26.553 INFO:teuthology.orchestra.run.smithi014.stdout:Setting up ceph (16.0.0-3402-g1ac5c74-1bionic) ... 2020-07-13T06:10:26.663 INFO:teuthology.orchestra.run.smithi014.stdout:Processing triggers for systemd (237-3ubuntu10.41) ... 2020-07-13T06:10:26.848 INFO:teuthology.orchestra.run.smithi014.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-07-13T06:10:28.108 INFO:teuthology.orchestra.run.smithi014.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-07-13T06:10:28.151 INFO:teuthology.orchestra.run.smithi014.stdout:ureadahead will be reprofiled on next reboot 2020-07-13T06:10:28.209 INFO:teuthology.orchestra.run.smithi014.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.2) ... 2020-07-13T06:10:28.652 INFO:teuthology.orchestra.run.smithi014.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-07-13T06:10:28.655 DEBUG:teuthology.parallel:result is None 2020-07-13T06:10:28.656 INFO:teuthology.orchestra.run.smithi017:> sudo lsb_release -is 2020-07-13T06:10:28.713 INFO:teuthology.orchestra.run.smithi017.stdout:Ubuntu 2020-07-13T06:10:28.719 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-07-13T06:10:28.720 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=1ac5c74502da6dc8ad965625df452eb42a846489 2020-07-13T06:10:28.900 INFO:teuthology.orchestra.run.smithi017:> dpkg-query -W -f '${Version}' ceph 2020-07-13T06:10:28.930 INFO:teuthology.orchestra.run.smithi017.stdout:16.0.0-3402-g1ac5c74-1bionic 2020-07-13T06:10:28.931 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-3402-g1ac5c74-1bionic 2020-07-13T06:10:28.932 INFO:teuthology.task.install:The correct ceph version 16.0.0-3402-g1ac5c74-1bionic is installed. 2020-07-13T06:10:28.933 INFO:teuthology.orchestra.run.smithi014:> sudo lsb_release -is 2020-07-13T06:10:28.992 INFO:teuthology.orchestra.run.smithi014.stdout:Ubuntu 2020-07-13T06:10:28.998 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-07-13T06:10:28.999 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=1ac5c74502da6dc8ad965625df452eb42a846489 2020-07-13T06:10:29.191 INFO:teuthology.orchestra.run.smithi014:> dpkg-query -W -f '${Version}' ceph 2020-07-13T06:10:29.217 INFO:teuthology.orchestra.run.smithi014.stdout:16.0.0-3402-g1ac5c74-1bionic 2020-07-13T06:10:29.218 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-3402-g1ac5c74-1bionic 2020-07-13T06:10:29.218 INFO:teuthology.task.install:The correct ceph version 16.0.0-3402-g1ac5c74-1bionic is installed. 2020-07-13T06:10:29.219 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-07-13T06:10:29.219 INFO:teuthology.orchestra.run.smithi017:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-07-13T06:10:29.236 INFO:teuthology.orchestra.run.smithi014:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-07-13T06:10:29.273 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-07-13T06:10:29.274 INFO:teuthology.orchestra.run.smithi017:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-07-13T06:10:29.298 INFO:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-07-13T06:10:29.353 INFO:teuthology.orchestra.run.smithi014:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-07-13T06:10:29.369 INFO:teuthology.orchestra.run.smithi014:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-07-13T06:10:29.426 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-07-13T06:10:29.434 INFO:teuthology.orchestra.run.smithi017:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-07-13T06:10:29.482 INFO:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-07-13T06:10:29.543 INFO:teuthology.orchestra.run.smithi014:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-07-13T06:10:29.589 INFO:teuthology.orchestra.run.smithi014:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-07-13T06:10:29.644 INFO:teuthology.run_tasks:Running task ceph... 2020-07-13T06:10:29.741 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-07-13T06:10:29.747 INFO:teuthology.orchestra.run.smithi014:> sudo chmod 777 /var/log/ceph 2020-07-13T06:10:29.750 INFO:teuthology.orchestra.run.smithi017:> sudo chmod 777 /var/log/ceph 2020-07-13T06:10:29.764 INFO:tasks.ceph:Disabling ceph logrotate... 2020-07-13T06:10:29.765 INFO:teuthology.orchestra.run.smithi014:> sudo rm -f -- /etc/logrotate.d/ceph 2020-07-13T06:10:29.808 INFO:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/logrotate.d/ceph 2020-07-13T06:10:29.823 INFO:tasks.ceph:Creating extra log directories... 2020-07-13T06:10:29.823 INFO:teuthology.orchestra.run.smithi014:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-07-13T06:10:29.864 INFO:teuthology.orchestra.run.smithi017:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-07-13T06:10:29.880 INFO:tasks.ceph:Creating ceph cluster c1... 2020-07-13T06:10:29.881 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-07-13T06:10:29.881 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2020-07-13_03:05:02-rgw-master-distro-basic-smithi/5218593', 'branch': 'master', 'description': 'rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '5218593', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2020-07-13_03:05:02-rgw-master-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'master'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '1ac5c74502da6dc8ad965625df452eb42a846489', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '1ac5c74502da6dc8ad965625df452eb42a846489'}}, 'rgw': {'compression type': 'random', 'frontend': 'civetweb'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'master', 'sha1': '1ac5c74502da6dc8ad965625df452eb42a846489'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 1000, 'repo': 'git://git.ceph.com/ceph.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '1ac5c74502da6dc8ad965625df452eb42a846489', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'master', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_master/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph.git', 'suite_sha1': '1ac5c74502da6dc8ad965625df452eb42a846489', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '1ac5c74502da6dc8ad965625df452eb42a846489', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.20773', 'targets': {'smithi017.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5Tg3xFvlmPJANVeH97ibsyYtT0l3duWosTCQdsiAKn2wAntdiIolQt5T5sNUSlPRj+sUH7eK5AF/+lijwzESp3GGzDRFrFoaWRbWOZeCGjTUUIKFFhlrf3hcXZCVQj9+Fz/1mUiZiM6dRnNEvvrFt7485R3OGrVpU6/QWvggV8Mqv/bLIerFbS1Z+3pD8jIFuepUKMV5IaHyXQrzmWESGmyR+cJYGu2CmjNEovrX3jzF9xCx+wFysKyorm0WzbveD4dQ+OyLixMa7xyO8/1i7nolBBmFltVOcYNuKeXTdRRZr+Hna3U/RutVzRuSPcOaXB6q39zmAeMC+6OIOuBz9', 'smithi014.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmQcdwKZF703ribUpzgoiJtFPNwO6YJpt3QdUhjrgrYsyMrNdTc6ePfhsPyQiUS7ch6f83BQanIliWLoIfxdHNHrOSdPtLtj569MND1/HJAnCfRm7Abc26MHr63Ej2ZrA3yfLTNz4p6na1Gm4wo0N3jBFuIP9kuXZeHoeTjIuU01e8Hcqmacb3ptSYlAaoqkblDgblem+kz6eF1yXiNCEydEgT2GKZ9djV/w8zYRrsWBu1b6J+Rq2Tn/Jp1XYFupOMo4Bpi6Ztm8GSlc0sMo0nETDec3Mwgpf/wcG65bYFVHkPPIvPl2u6VZ5yE0qaO+zPFz409WyM4gRPRkZV5lPL'}} 2020-07-13T06:10:29.882 INFO:teuthology.orchestra.run.smithi014:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-07-13T06:10:29.924 INFO:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-07-13T06:10:29.932 INFO:teuthology.orchestra.run.smithi014:> sudo install -d -m0777 -- /var/run/ceph 2020-07-13T06:10:29.973 INFO:teuthology.orchestra.run.smithi017:> sudo install -d -m0777 -- /var/run/ceph 2020-07-13T06:10:30.042 DEBUG:teuthology.orchestra.remote:smithi017:/scratch_devs is 72B 2020-07-13T06:10:30.054 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-07-13T06:10:30.054 INFO:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-07-13T06:10:30.061 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-07-13T06:10:30.061 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-13T06:10:30.061 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 342 Links: 1 2020-07-13T06:10:30.062 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-13T06:10:30.062 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2020-07-13 06:08:02.880267877 +0000 2020-07-13T06:10:30.062 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2020-07-13 06:08:02.700267881 +0000 2020-07-13T06:10:30.062 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2020-07-13 06:08:02.700267881 +0000 2020-07-13T06:10:30.062 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2020-07-13T06:10:30.070 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2020-07-13T06:10:30.071 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2020-07-13T06:10:30.071 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000373265 s, 1.4 MB/s 2020-07-13T06:10:30.075 INFO:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-07-13T06:10:30.122 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-07-13T06:10:30.122 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-13T06:10:30.123 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 351 Links: 1 2020-07-13T06:10:30.123 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-13T06:10:30.123 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2020-07-13 06:08:03.024267874 +0000 2020-07-13T06:10:30.123 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2020-07-13 06:08:03.020267874 +0000 2020-07-13T06:10:30.124 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2020-07-13 06:08:03.020267874 +0000 2020-07-13T06:10:30.124 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2020-07-13T06:10:30.132 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2020-07-13T06:10:30.135 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2020-07-13T06:10:30.137 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.00037201 s, 1.4 MB/s 2020-07-13T06:10:30.140 INFO:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-07-13T06:10:30.191 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-07-13T06:10:30.192 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-13T06:10:30.192 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 357 Links: 1 2020-07-13T06:10:30.192 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-13T06:10:30.193 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2020-07-13 06:08:03.400267866 +0000 2020-07-13T06:10:30.193 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2020-07-13 06:08:03.280267868 +0000 2020-07-13T06:10:30.193 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2020-07-13 06:08:03.280267868 +0000 2020-07-13T06:10:30.193 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2020-07-13T06:10:30.199 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2020-07-13T06:10:30.200 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2020-07-13T06:10:30.200 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000380224 s, 1.3 MB/s 2020-07-13T06:10:30.206 INFO:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-07-13T06:10:30.254 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-07-13T06:10:30.254 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-13T06:10:30.254 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 363 Links: 1 2020-07-13T06:10:30.255 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-13T06:10:30.255 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2020-07-13 06:08:03.684267859 +0000 2020-07-13T06:10:30.255 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2020-07-13 06:08:03.532267863 +0000 2020-07-13T06:10:30.255 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2020-07-13 06:08:03.532267863 +0000 2020-07-13T06:10:30.256 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2020-07-13T06:10:30.263 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2020-07-13T06:10:30.263 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2020-07-13T06:10:30.263 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000403033 s, 1.3 MB/s 2020-07-13T06:10:30.266 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-13T06:10:30.267 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi017.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-07-13T06:10:30.267 INFO:tasks.ceph:Generating config... 2020-07-13T06:10:30.269 INFO:tasks.ceph:[client] debug rgw = 20 2020-07-13T06:10:30.269 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-07-13T06:10:30.269 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-07-13T06:10:30.270 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-07-13T06:10:30.270 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-07-13T06:10:30.272 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-07-13T06:10:30.272 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-07-13T06:10:30.273 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-07-13T06:10:30.273 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-07-13T06:10:30.273 INFO:tasks.ceph:[client] setgroup = ceph 2020-07-13T06:10:30.274 INFO:tasks.ceph:[client] setuser = ceph 2020-07-13T06:10:30.274 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-07-13T06:10:30.274 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-07-13T06:10:30.275 INFO:tasks.ceph:[mgr] debug ms = 1 2020-07-13T06:10:30.275 INFO:tasks.ceph:[mon] debug mon = 20 2020-07-13T06:10:30.276 INFO:tasks.ceph:[mon] debug ms = 1 2020-07-13T06:10:30.276 INFO:tasks.ceph:[mon] debug paxos = 20 2020-07-13T06:10:30.276 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-07-13T06:10:30.277 INFO:tasks.ceph:[osd] debug filestore = 20 2020-07-13T06:10:30.278 INFO:tasks.ceph:[osd] debug journal = 20 2020-07-13T06:10:30.278 INFO:tasks.ceph:[osd] debug ms = 20 2020-07-13T06:10:30.278 INFO:tasks.ceph:[osd] debug osd = 25 2020-07-13T06:10:30.279 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-07-13T06:10:30.279 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-07-13T06:10:30.279 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-07-13T06:10:30.279 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-07-13T06:10:30.280 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-07-13T06:10:30.280 INFO:tasks.ceph:Setting up c1.mon.a... 2020-07-13T06:10:30.281 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2020-07-13T06:10:30.341 INFO:teuthology.orchestra.run.smithi017.stdout:creating /etc/ceph/c1.keyring 2020-07-13T06:10:30.344 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2020-07-13T06:10:30.422 INFO:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-07-13T06:10:30.482 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.17')] 2020-07-13T06:10:30.482 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.17 --print /home/ubuntu/cephtest/c1.monmap 2020-07-13T06:10:30.554 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2020-07-13T06:10:30.554 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: generated fsid a3ab2581-a560-49c3-bc72-913c25ea7119 2020-07-13T06:10:30.554 INFO:teuthology.orchestra.run.smithi017.stdout:epoch 0 2020-07-13T06:10:30.555 INFO:teuthology.orchestra.run.smithi017.stdout:fsid a3ab2581-a560-49c3-bc72-913c25ea7119 2020-07-13T06:10:30.555 INFO:teuthology.orchestra.run.smithi017.stdout:last_changed 2020-07-13T06:10:30.555291+0000 2020-07-13T06:10:30.555 INFO:teuthology.orchestra.run.smithi017.stdout:created 2020-07-13T06:10:30.555291+0000 2020-07-13T06:10:30.555 INFO:teuthology.orchestra.run.smithi017.stdout:min_mon_release 0 (unknown) 2020-07-13T06:10:30.555 INFO:teuthology.orchestra.run.smithi017.stdout:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2020-07-13T06:10:30.556 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2020-07-13T06:10:30.557 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID a3ab2581-a560-49c3-bc72-913c25ea7119... 2020-07-13T06:10:30.558 INFO:teuthology.orchestra.run.smithi014:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-07-13T06:10:30.561 INFO:teuthology.orchestra.run.smithi017:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-07-13T06:10:30.633 INFO:teuthology.orchestra.run.smithi017.stdout:[global] 2020-07-13T06:10:30.634 INFO:teuthology.orchestra.run.smithi017.stdout: chdir = "" 2020-07-13T06:10:30.634 INFO:teuthology.orchestra.run.smithi017.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-07-13T06:10:30.635 INFO:teuthology.orchestra.run.smithi017.stdout: auth supported = cephx 2020-07-13T06:10:30.635 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.635 INFO:teuthology.orchestra.run.smithi017.stdout: filestore xattr use omap = true 2020-07-13T06:10:30.635 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.635 INFO:teuthology.orchestra.run.smithi017.stdout: mon clock drift allowed = 1.000 2020-07-13T06:10:30.636 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.636 INFO:teuthology.orchestra.run.smithi017.stdout: osd crush chooseleaf type = 0 2020-07-13T06:10:30.636 INFO:teuthology.orchestra.run.smithi017.stdout: auth debug = true 2020-07-13T06:10:30.636 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.636 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on old message = true 2020-07-13T06:10:30.637 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on bug = true 2020-07-13T06:10:30.637 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.637 INFO:teuthology.orchestra.run.smithi017.stdout: mon max pg per osd = 10000 # >= luminous 2020-07-13T06:10:30.637 INFO:teuthology.orchestra.run.smithi017.stdout: mon pg warn max object skew = 0 2020-07-13T06:10:30.638 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.638 INFO:teuthology.orchestra.run.smithi017.stdout: # disable pg_autoscaler by default for new pools 2020-07-13T06:10:30.638 INFO:teuthology.orchestra.run.smithi017.stdout: osd_pool_default_pg_autoscale_mode = off 2020-07-13T06:10:30.638 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.638 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default size = 2 2020-07-13T06:10:30.639 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.639 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow primary affinity = true 2020-07-13T06:10:30.639 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow pg remap = true 2020-07-13T06:10:30.639 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on legacy crush tunables = false 2020-07-13T06:10:30.639 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on crush straw calc version zero = false 2020-07-13T06:10:30.640 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on no sortbitwise = false 2020-07-13T06:10:30.640 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on osd down out interval zero = false 2020-07-13T06:10:30.640 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on too few osds = false 2020-07-13T06:10:30.640 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-07-13T06:10:30.640 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_no_redundancy = false 2020-07-13T06:10:30.641 INFO:teuthology.orchestra.run.smithi017.stdout: mon_allow_pool_size_one = true 2020-07-13T06:10:30.641 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.641 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-07-13T06:10:30.641 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.641 INFO:teuthology.orchestra.run.smithi017.stdout: osd default data pool replay window = 5 2020-07-13T06:10:30.642 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.642 INFO:teuthology.orchestra.run.smithi017.stdout: mon allow pool delete = true 2020-07-13T06:10:30.643 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.643 INFO:teuthology.orchestra.run.smithi017.stdout: mon cluster log file level = debug 2020-07-13T06:10:30.643 INFO:teuthology.orchestra.run.smithi017.stdout: debug asserts on shutdown = true 2020-07-13T06:10:30.643 INFO:teuthology.orchestra.run.smithi017.stdout: mon host = 172.21.15.17 2020-07-13T06:10:30.643 INFO:teuthology.orchestra.run.smithi017.stdout: osd heartbeat grace = 40 2020-07-13T06:10:30.644 INFO:teuthology.orchestra.run.smithi017.stdout: fsid = a3ab2581-a560-49c3-bc72-913c25ea7119 2020-07-13T06:10:30.644 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.644 INFO:teuthology.orchestra.run.smithi017.stdout:[osd] 2020-07-13T06:10:30.644 INFO:teuthology.orchestra.run.smithi017.stdout: osd journal size = 100 2020-07-13T06:10:30.645 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.645 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub load threshold = 5.0 2020-07-13T06:10:30.645 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub max interval = 600 2020-07-13T06:10:30.645 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.646 INFO:teuthology.orchestra.run.smithi017.stdout: osd recover clone overlap = true 2020-07-13T06:10:30.646 INFO:teuthology.orchestra.run.smithi017.stdout: osd recovery max chunk = 1048576 2020-07-13T06:10:30.646 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.646 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug shutdown = true 2020-07-13T06:10:30.647 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug op order = true 2020-07-13T06:10:30.647 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug verify stray on activate = true 2020-07-13T06:10:30.647 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.647 INFO:teuthology.orchestra.run.smithi017.stdout: osd open classes on start = true 2020-07-13T06:10:30.648 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug pg log writeout = true 2020-07-13T06:10:30.648 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.648 INFO:teuthology.orchestra.run.smithi017.stdout: osd deep scrub update digest min age = 30 2020-07-13T06:10:30.648 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.648 INFO:teuthology.orchestra.run.smithi017.stdout: osd map max advance = 10 2020-07-13T06:10:30.649 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.650 INFO:teuthology.orchestra.run.smithi017.stdout: journal zero on create = true 2020-07-13T06:10:30.650 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.650 INFO:teuthology.orchestra.run.smithi017.stdout: filestore ondisk finisher threads = 3 2020-07-13T06:10:30.651 INFO:teuthology.orchestra.run.smithi017.stdout: filestore apply finisher threads = 3 2020-07-13T06:10:30.651 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.651 INFO:teuthology.orchestra.run.smithi017.stdout: bdev debug aio = true 2020-07-13T06:10:30.651 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug misdirected ops = true 2020-07-13T06:10:30.651 INFO:teuthology.orchestra.run.smithi017.stdout: debug filestore = 20 2020-07-13T06:10:30.652 INFO:teuthology.orchestra.run.smithi017.stdout: debug journal = 20 2020-07-13T06:10:30.652 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 20 2020-07-13T06:10:30.652 INFO:teuthology.orchestra.run.smithi017.stdout: debug osd = 25 2020-07-13T06:10:30.652 INFO:teuthology.orchestra.run.smithi017.stdout: osd fast shutdown = False 2020-07-13T06:10:30.652 INFO:teuthology.orchestra.run.smithi017.stdout: rocksdb delete range threshold = 0 2020-07-13T06:10:30.653 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.653 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr] 2020-07-13T06:10:30.653 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2020-07-13T06:10:30.653 INFO:teuthology.orchestra.run.smithi017.stdout: debug mgr = 20 2020-07-13T06:10:30.653 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2020-07-13T06:10:30.654 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2020-07-13T06:10:30.654 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2020-07-13T06:10:30.654 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2020-07-13T06:10:30.654 INFO:teuthology.orchestra.run.smithi017.stdout: mgr/telemetry/nag = false 2020-07-13T06:10:30.654 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.655 INFO:teuthology.orchestra.run.smithi017.stdout:[mon] 2020-07-13T06:10:30.655 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2020-07-13T06:10:30.655 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2020-07-13T06:10:30.656 INFO:teuthology.orchestra.run.smithi017.stdout: debug paxos = 20 2020-07-13T06:10:30.656 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2020-07-13T06:10:30.656 INFO:teuthology.orchestra.run.smithi017.stdout: mon data avail warn = 5 2020-07-13T06:10:30.656 INFO:teuthology.orchestra.run.smithi017.stdout: mon mgr mkfs grace = 240 2020-07-13T06:10:30.656 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2020-07-13T06:10:30.656 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd reporter subtree level = osd 2020-07-13T06:10:30.657 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd prime pg temp = true 2020-07-13T06:10:30.657 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2020-07-13T06:10:30.657 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd crush smoke test = False 2020-07-13T06:10:30.657 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:30.657 INFO:teuthology.orchestra.run.smithi017.stdout:[client] 2020-07-13T06:10:30.658 INFO:teuthology.orchestra.run.smithi017.stdout: rgw cache enabled = true 2020-07-13T06:10:30.658 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable ops log = true 2020-07-13T06:10:30.658 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable usage log = true 2020-07-13T06:10:30.658 INFO:teuthology.orchestra.run.smithi017.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-07-13T06:10:30.658 INFO:teuthology.orchestra.run.smithi017.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-07-13T06:10:30.659 INFO:teuthology.orchestra.run.smithi017.stdout: debug rgw = 20 2020-07-13T06:10:30.659 INFO:teuthology.orchestra.run.smithi017.stdout: rgw crypt require ssl = False 2020-07-13T06:10:30.659 INFO:teuthology.orchestra.run.smithi017.stdout: rgw crypt s3 kms backend = testing 2020-07-13T06:10:30.659 INFO:teuthology.orchestra.run.smithi017.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-07-13T06:10:30.660 INFO:teuthology.orchestra.run.smithi017.stdout: rgw curl low speed time = 300 2020-07-13T06:10:30.660 INFO:teuthology.orchestra.run.smithi017.stdout: rgw data log num shards = 4 2020-07-13T06:10:30.660 INFO:teuthology.orchestra.run.smithi017.stdout: rgw md log max shards = 4 2020-07-13T06:10:30.660 INFO:teuthology.orchestra.run.smithi017.stdout: rgw sync log trim interval = 0 2020-07-13T06:10:30.660 INFO:teuthology.orchestra.run.smithi017.stdout: rgw sync obj etag verify = True 2020-07-13T06:10:30.661 INFO:teuthology.orchestra.run.smithi017.stdout: setgroup = ceph 2020-07-13T06:10:30.661 INFO:teuthology.orchestra.run.smithi017.stdout: setuser = ceph 2020-07-13T06:10:30.661 INFO:teuthology.orchestra.run.smithi017.stdout:[mon.a] 2020-07-13T06:10:30.661 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.0] 2020-07-13T06:10:30.661 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_omap_entries_per_request = 10 2020-07-13T06:10:30.661 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.1] 2020-07-13T06:10:30.662 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_omap_entries_per_request = 1000 2020-07-13T06:10:30.662 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.2] 2020-07-13T06:10:30.662 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_omap_entries_per_request = 10000 2020-07-13T06:10:30.673 INFO:teuthology.orchestra.run.smithi014.stdout:[global] 2020-07-13T06:10:30.673 INFO:teuthology.orchestra.run.smithi014.stdout: chdir = "" 2020-07-13T06:10:30.673 INFO:teuthology.orchestra.run.smithi014.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-07-13T06:10:30.673 INFO:teuthology.orchestra.run.smithi014.stdout: auth supported = cephx 2020-07-13T06:10:30.674 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.674 INFO:teuthology.orchestra.run.smithi014.stdout: filestore xattr use omap = true 2020-07-13T06:10:30.674 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.674 INFO:teuthology.orchestra.run.smithi014.stdout: mon clock drift allowed = 1.000 2020-07-13T06:10:30.674 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.675 INFO:teuthology.orchestra.run.smithi014.stdout: osd crush chooseleaf type = 0 2020-07-13T06:10:30.675 INFO:teuthology.orchestra.run.smithi014.stdout: auth debug = true 2020-07-13T06:10:30.675 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.675 INFO:teuthology.orchestra.run.smithi014.stdout: ms die on old message = true 2020-07-13T06:10:30.675 INFO:teuthology.orchestra.run.smithi014.stdout: ms die on bug = true 2020-07-13T06:10:30.676 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.676 INFO:teuthology.orchestra.run.smithi014.stdout: mon max pg per osd = 10000 # >= luminous 2020-07-13T06:10:30.676 INFO:teuthology.orchestra.run.smithi014.stdout: mon pg warn max object skew = 0 2020-07-13T06:10:30.676 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.677 INFO:teuthology.orchestra.run.smithi014.stdout: # disable pg_autoscaler by default for new pools 2020-07-13T06:10:30.677 INFO:teuthology.orchestra.run.smithi014.stdout: osd_pool_default_pg_autoscale_mode = off 2020-07-13T06:10:30.677 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.677 INFO:teuthology.orchestra.run.smithi014.stdout: osd pool default size = 2 2020-07-13T06:10:30.677 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.678 INFO:teuthology.orchestra.run.smithi014.stdout: mon osd allow primary affinity = true 2020-07-13T06:10:30.678 INFO:teuthology.orchestra.run.smithi014.stdout: mon osd allow pg remap = true 2020-07-13T06:10:30.678 INFO:teuthology.orchestra.run.smithi014.stdout: mon warn on legacy crush tunables = false 2020-07-13T06:10:30.678 INFO:teuthology.orchestra.run.smithi014.stdout: mon warn on crush straw calc version zero = false 2020-07-13T06:10:30.678 INFO:teuthology.orchestra.run.smithi014.stdout: mon warn on no sortbitwise = false 2020-07-13T06:10:30.679 INFO:teuthology.orchestra.run.smithi014.stdout: mon warn on osd down out interval zero = false 2020-07-13T06:10:30.679 INFO:teuthology.orchestra.run.smithi014.stdout: mon warn on too few osds = false 2020-07-13T06:10:30.679 INFO:teuthology.orchestra.run.smithi014.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-07-13T06:10:30.679 INFO:teuthology.orchestra.run.smithi014.stdout: mon_warn_on_pool_no_redundancy = false 2020-07-13T06:10:30.680 INFO:teuthology.orchestra.run.smithi014.stdout: mon_allow_pool_size_one = true 2020-07-13T06:10:30.680 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.680 INFO:teuthology.orchestra.run.smithi014.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-07-13T06:10:30.680 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.681 INFO:teuthology.orchestra.run.smithi014.stdout: osd default data pool replay window = 5 2020-07-13T06:10:30.681 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.681 INFO:teuthology.orchestra.run.smithi014.stdout: mon allow pool delete = true 2020-07-13T06:10:30.681 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.682 INFO:teuthology.orchestra.run.smithi014.stdout: mon cluster log file level = debug 2020-07-13T06:10:30.682 INFO:teuthology.orchestra.run.smithi014.stdout: debug asserts on shutdown = true 2020-07-13T06:10:30.682 INFO:teuthology.orchestra.run.smithi014.stdout: mon host = 172.21.15.17 2020-07-13T06:10:30.682 INFO:teuthology.orchestra.run.smithi014.stdout: osd heartbeat grace = 40 2020-07-13T06:10:30.682 INFO:teuthology.orchestra.run.smithi014.stdout: fsid = a3ab2581-a560-49c3-bc72-913c25ea7119 2020-07-13T06:10:30.682 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.683 INFO:teuthology.orchestra.run.smithi014.stdout:[osd] 2020-07-13T06:10:30.683 INFO:teuthology.orchestra.run.smithi014.stdout: osd journal size = 100 2020-07-13T06:10:30.683 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.683 INFO:teuthology.orchestra.run.smithi014.stdout: osd scrub load threshold = 5.0 2020-07-13T06:10:30.683 INFO:teuthology.orchestra.run.smithi014.stdout: osd scrub max interval = 600 2020-07-13T06:10:30.684 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.684 INFO:teuthology.orchestra.run.smithi014.stdout: osd recover clone overlap = true 2020-07-13T06:10:30.684 INFO:teuthology.orchestra.run.smithi014.stdout: osd recovery max chunk = 1048576 2020-07-13T06:10:30.684 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.684 INFO:teuthology.orchestra.run.smithi014.stdout: osd debug shutdown = true 2020-07-13T06:10:30.685 INFO:teuthology.orchestra.run.smithi014.stdout: osd debug op order = true 2020-07-13T06:10:30.685 INFO:teuthology.orchestra.run.smithi014.stdout: osd debug verify stray on activate = true 2020-07-13T06:10:30.685 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.685 INFO:teuthology.orchestra.run.smithi014.stdout: osd open classes on start = true 2020-07-13T06:10:30.685 INFO:teuthology.orchestra.run.smithi014.stdout: osd debug pg log writeout = true 2020-07-13T06:10:30.685 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.686 INFO:teuthology.orchestra.run.smithi014.stdout: osd deep scrub update digest min age = 30 2020-07-13T06:10:30.686 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.686 INFO:teuthology.orchestra.run.smithi014.stdout: osd map max advance = 10 2020-07-13T06:10:30.687 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.687 INFO:teuthology.orchestra.run.smithi014.stdout: journal zero on create = true 2020-07-13T06:10:30.687 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.687 INFO:teuthology.orchestra.run.smithi014.stdout: filestore ondisk finisher threads = 3 2020-07-13T06:10:30.687 INFO:teuthology.orchestra.run.smithi014.stdout: filestore apply finisher threads = 3 2020-07-13T06:10:30.688 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.688 INFO:teuthology.orchestra.run.smithi014.stdout: bdev debug aio = true 2020-07-13T06:10:30.688 INFO:teuthology.orchestra.run.smithi014.stdout: osd debug misdirected ops = true 2020-07-13T06:10:30.688 INFO:teuthology.orchestra.run.smithi014.stdout: debug filestore = 20 2020-07-13T06:10:30.688 INFO:teuthology.orchestra.run.smithi014.stdout: debug journal = 20 2020-07-13T06:10:30.689 INFO:teuthology.orchestra.run.smithi014.stdout: debug ms = 20 2020-07-13T06:10:30.689 INFO:teuthology.orchestra.run.smithi014.stdout: debug osd = 25 2020-07-13T06:10:30.689 INFO:teuthology.orchestra.run.smithi014.stdout: osd fast shutdown = False 2020-07-13T06:10:30.689 INFO:teuthology.orchestra.run.smithi014.stdout: rocksdb delete range threshold = 0 2020-07-13T06:10:30.689 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.690 INFO:teuthology.orchestra.run.smithi014.stdout:[mgr] 2020-07-13T06:10:30.690 INFO:teuthology.orchestra.run.smithi014.stdout: debug ms = 1 2020-07-13T06:10:30.690 INFO:teuthology.orchestra.run.smithi014.stdout: debug mgr = 20 2020-07-13T06:10:30.690 INFO:teuthology.orchestra.run.smithi014.stdout: debug mon = 20 2020-07-13T06:10:30.690 INFO:teuthology.orchestra.run.smithi014.stdout: debug auth = 20 2020-07-13T06:10:30.690 INFO:teuthology.orchestra.run.smithi014.stdout: mon reweight min pgs per osd = 4 2020-07-13T06:10:30.691 INFO:teuthology.orchestra.run.smithi014.stdout: mon reweight min bytes per osd = 10 2020-07-13T06:10:30.691 INFO:teuthology.orchestra.run.smithi014.stdout: mgr/telemetry/nag = false 2020-07-13T06:10:30.691 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.691 INFO:teuthology.orchestra.run.smithi014.stdout:[mon] 2020-07-13T06:10:30.691 INFO:teuthology.orchestra.run.smithi014.stdout: debug ms = 1 2020-07-13T06:10:30.692 INFO:teuthology.orchestra.run.smithi014.stdout: debug mon = 20 2020-07-13T06:10:30.692 INFO:teuthology.orchestra.run.smithi014.stdout: debug paxos = 20 2020-07-13T06:10:30.692 INFO:teuthology.orchestra.run.smithi014.stdout: debug auth = 20 2020-07-13T06:10:30.692 INFO:teuthology.orchestra.run.smithi014.stdout: mon data avail warn = 5 2020-07-13T06:10:30.692 INFO:teuthology.orchestra.run.smithi014.stdout: mon mgr mkfs grace = 240 2020-07-13T06:10:30.693 INFO:teuthology.orchestra.run.smithi014.stdout: mon reweight min pgs per osd = 4 2020-07-13T06:10:30.693 INFO:teuthology.orchestra.run.smithi014.stdout: mon osd reporter subtree level = osd 2020-07-13T06:10:30.693 INFO:teuthology.orchestra.run.smithi014.stdout: mon osd prime pg temp = true 2020-07-13T06:10:30.693 INFO:teuthology.orchestra.run.smithi014.stdout: mon reweight min bytes per osd = 10 2020-07-13T06:10:30.693 INFO:teuthology.orchestra.run.smithi014.stdout: mon osd crush smoke test = False 2020-07-13T06:10:30.694 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:30.694 INFO:teuthology.orchestra.run.smithi014.stdout:[client] 2020-07-13T06:10:30.694 INFO:teuthology.orchestra.run.smithi014.stdout: rgw cache enabled = true 2020-07-13T06:10:30.694 INFO:teuthology.orchestra.run.smithi014.stdout: rgw enable ops log = true 2020-07-13T06:10:30.694 INFO:teuthology.orchestra.run.smithi014.stdout: rgw enable usage log = true 2020-07-13T06:10:30.695 INFO:teuthology.orchestra.run.smithi014.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-07-13T06:10:30.695 INFO:teuthology.orchestra.run.smithi014.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-07-13T06:10:30.695 INFO:teuthology.orchestra.run.smithi014.stdout: debug rgw = 20 2020-07-13T06:10:30.695 INFO:teuthology.orchestra.run.smithi014.stdout: rgw crypt require ssl = False 2020-07-13T06:10:30.695 INFO:teuthology.orchestra.run.smithi014.stdout: rgw crypt s3 kms backend = testing 2020-07-13T06:10:30.695 INFO:teuthology.orchestra.run.smithi014.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-07-13T06:10:30.696 INFO:teuthology.orchestra.run.smithi014.stdout: rgw curl low speed time = 300 2020-07-13T06:10:30.696 INFO:teuthology.orchestra.run.smithi014.stdout: rgw data log num shards = 4 2020-07-13T06:10:30.696 INFO:teuthology.orchestra.run.smithi014.stdout: rgw md log max shards = 4 2020-07-13T06:10:30.697 INFO:teuthology.orchestra.run.smithi014.stdout: rgw sync log trim interval = 0 2020-07-13T06:10:30.697 INFO:teuthology.orchestra.run.smithi014.stdout: rgw sync obj etag verify = True 2020-07-13T06:10:30.697 INFO:teuthology.orchestra.run.smithi014.stdout: setgroup = ceph 2020-07-13T06:10:30.697 INFO:teuthology.orchestra.run.smithi014.stdout: setuser = ceph 2020-07-13T06:10:30.697 INFO:teuthology.orchestra.run.smithi014.stdout:[mon.a] 2020-07-13T06:10:30.698 INFO:teuthology.orchestra.run.smithi014.stdout:[osd.0] 2020-07-13T06:10:30.698 INFO:teuthology.orchestra.run.smithi014.stdout: osd_max_omap_entries_per_request = 10 2020-07-13T06:10:30.698 INFO:teuthology.orchestra.run.smithi014.stdout:[osd.1] 2020-07-13T06:10:30.698 INFO:teuthology.orchestra.run.smithi014.stdout: osd_max_omap_entries_per_request = 1000 2020-07-13T06:10:30.698 INFO:teuthology.orchestra.run.smithi014.stdout:[osd.2] 2020-07-13T06:10:30.699 INFO:teuthology.orchestra.run.smithi014.stdout: osd_max_omap_entries_per_request = 10000 2020-07-13T06:10:30.700 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2020-07-13T06:10:30.700 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2020-07-13T06:10:30.753 INFO:tasks.ceph:Copying monmap to all nodes... 2020-07-13T06:10:30.760 DEBUG:teuthology.orchestra.remote:smithi017:/etc/ceph/c1.keyring is 206B 2020-07-13T06:10:30.778 DEBUG:teuthology.orchestra.remote:smithi017:/home/ubuntu/cephtest/c1.monmap is 179B 2020-07-13T06:10:30.794 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi017.front.sepia.ceph.com 2020-07-13T06:10:30.795 INFO:teuthology.orchestra.run.smithi017:> sudo sh -c 'cat > /etc/ceph/c1.keyring' && sudo chmod 0644 /etc/ceph/c1.keyring 2020-07-13T06:10:30.851 INFO:teuthology.orchestra.run.smithi017:> cat > /home/ubuntu/cephtest/c1.monmap 2020-07-13T06:10:30.941 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi014.front.sepia.ceph.com 2020-07-13T06:10:30.941 INFO:teuthology.orchestra.run.smithi014:> sudo sh -c 'cat > /etc/ceph/c1.keyring' && sudo chmod 0644 /etc/ceph/c1.keyring 2020-07-13T06:10:31.003 INFO:teuthology.orchestra.run.smithi014:> cat > /home/ubuntu/cephtest/c1.monmap 2020-07-13T06:10:31.092 INFO:tasks.ceph:Setting up mon nodes... 2020-07-13T06:10:31.093 INFO:tasks.ceph:Setting up mgr nodes... 2020-07-13T06:10:31.094 INFO:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2020-07-13T06:10:31.138 INFO:teuthology.orchestra.run.smithi017.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2020-07-13T06:10:31.139 INFO:tasks.ceph:Setting up mds nodes... 2020-07-13T06:10:31.141 INFO:tasks.ceph_client:Setting up client nodes... 2020-07-13T06:10:31.142 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2020-07-13T06:10:31.216 INFO:teuthology.orchestra.run.smithi017.stdout:creating /etc/ceph/c1.client.0.keyring 2020-07-13T06:10:31.229 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2020-07-13T06:10:31.306 INFO:teuthology.orchestra.run.smithi017.stdout:creating /etc/ceph/c1.client.1.keyring 2020-07-13T06:10:31.319 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-07-13T06:10:31.321 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi017.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-07-13T06:10:31.325 INFO:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2020-07-13T06:10:31.389 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-13T06:10:31.392 INFO:tasks.ceph:role: c1.osd.0 2020-07-13T06:10:31.394 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi017.front.sepia.ceph.com 2020-07-13T06:10:31.395 INFO:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-07-13T06:10:31.911 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-07-13T06:10:31.911 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2020-07-13T06:10:31.911 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-07-13T06:10:31.912 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-07-13T06:10:31.912 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2020-07-13T06:10:31.912 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-07-13T06:10:31.912 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-07-13T06:10:31.912 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-07-13T06:10:31.912 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-07-13T06:10:31.914 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2020-07-13T06:10:31.914 INFO:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2020-07-13T06:10:31.989 INFO:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2020-07-13T06:10:32.000 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: /sbin/restorecon: command not found 2020-07-13T06:10:32.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-07-13T06:10:32.001 INFO:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2020-07-13T06:10:32.058 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-13T06:10:32.058 INFO:tasks.ceph:role: c1.osd.1 2020-07-13T06:10:32.059 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi017.front.sepia.ceph.com 2020-07-13T06:10:32.060 INFO:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-07-13T06:10:32.565 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-07-13T06:10:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2020-07-13T06:10:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-07-13T06:10:32.566 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-07-13T06:10:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2020-07-13T06:10:32.566 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-07-13T06:10:32.567 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-07-13T06:10:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-07-13T06:10:32.567 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-07-13T06:10:32.568 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2020-07-13T06:10:32.568 INFO:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2020-07-13T06:10:32.632 INFO:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2020-07-13T06:10:32.645 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: /sbin/restorecon: command not found 2020-07-13T06:10:32.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-07-13T06:10:32.646 INFO:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2020-07-13T06:10:32.700 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-13T06:10:32.701 INFO:tasks.ceph:role: c1.osd.2 2020-07-13T06:10:32.701 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi017.front.sepia.ceph.com 2020-07-13T06:10:32.702 INFO:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-07-13T06:10:33.209 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-07-13T06:10:33.209 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2020-07-13T06:10:33.209 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-07-13T06:10:33.210 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-07-13T06:10:33.210 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2020-07-13T06:10:33.210 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-07-13T06:10:33.210 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-07-13T06:10:33.210 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-07-13T06:10:33.211 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-07-13T06:10:33.212 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2020-07-13T06:10:33.212 INFO:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2020-07-13T06:10:33.272 INFO:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2020-07-13T06:10:33.285 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: /sbin/restorecon: command not found 2020-07-13T06:10:33.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-07-13T06:10:33.286 INFO:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2020-07-13T06:10:33.402 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:10:33.399+0000 7f641f4f8d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2020-07-13T06:10:33.404 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:10:33.403+0000 7f641f4f8d80 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2020-07-13T06:10:33.405 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:10:33.403+0000 7f641f4f8d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-07-13T06:10:33.405 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:10:33.403+0000 7f641f4f8d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-07-13T06:10:33.405 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:10:33.403+0000 7f641f4f8d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-07-13T06:10:33.406 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:10:33.403+0000 7f641f4f8d80 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2020-07-13T06:10:35.343 INFO:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2020-07-13T06:10:35.356 INFO:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2020-07-13T06:10:35.438 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:10:35.435+0000 7f36c4431d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2020-07-13T06:10:35.438 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:10:35.435+0000 7f36c4431d80 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2020-07-13T06:10:35.438 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:10:35.435+0000 7f36c4431d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-07-13T06:10:35.439 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:10:35.435+0000 7f36c4431d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-07-13T06:10:35.439 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:10:35.435+0000 7f36c4431d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-07-13T06:10:35.439 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:10:35.435+0000 7f36c4431d80 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2020-07-13T06:10:37.367 INFO:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2020-07-13T06:10:37.422 INFO:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2020-07-13T06:10:37.502 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:10:37.499+0000 7fde92759d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2020-07-13T06:10:37.502 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:10:37.499+0000 7fde92759d80 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2020-07-13T06:10:37.503 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:10:37.499+0000 7fde92759d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-07-13T06:10:37.503 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:10:37.499+0000 7fde92759d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-07-13T06:10:37.503 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:10:37.499+0000 7fde92759d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-07-13T06:10:37.504 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:10:37.499+0000 7fde92759d80 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2020-07-13T06:10:39.436 INFO:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2020-07-13T06:10:39.494 INFO:tasks.ceph:Reading keys from all nodes... 2020-07-13T06:10:39.495 INFO:teuthology.orchestra.run.smithi017:> mktemp 2020-07-13T06:10:39.541 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.mqk3PIC7QZ 2020-07-13T06:10:39.541 INFO:teuthology.orchestra.run.smithi017:> sudo cp /var/lib/ceph/mgr/c1-x/keyring /tmp/tmp.mqk3PIC7QZ 2020-07-13T06:10:39.598 INFO:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.mqk3PIC7QZ 2020-07-13T06:10:39.707 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.mqk3PIC7QZ is 56B 2020-07-13T06:10:39.758 INFO:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.mqk3PIC7QZ 2020-07-13T06:10:39.766 INFO:teuthology.orchestra.run.smithi017:> mktemp 2020-07-13T06:10:39.813 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.Fn6Z43ID4A 2020-07-13T06:10:39.814 INFO:teuthology.orchestra.run.smithi017:> sudo cp /var/lib/ceph/osd/c1-0/keyring /tmp/tmp.Fn6Z43ID4A 2020-07-13T06:10:39.871 INFO:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.Fn6Z43ID4A 2020-07-13T06:10:39.983 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.Fn6Z43ID4A is 56B 2020-07-13T06:10:40.038 INFO:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.Fn6Z43ID4A 2020-07-13T06:10:40.046 INFO:teuthology.orchestra.run.smithi017:> mktemp 2020-07-13T06:10:40.093 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.8qMTuNrgQZ 2020-07-13T06:10:40.094 INFO:teuthology.orchestra.run.smithi017:> sudo cp /var/lib/ceph/osd/c1-1/keyring /tmp/tmp.8qMTuNrgQZ 2020-07-13T06:10:40.151 INFO:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.8qMTuNrgQZ 2020-07-13T06:10:40.256 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.8qMTuNrgQZ is 56B 2020-07-13T06:10:40.266 INFO:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.8qMTuNrgQZ 2020-07-13T06:10:40.273 INFO:teuthology.orchestra.run.smithi017:> mktemp 2020-07-13T06:10:40.325 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.NTaPY6jtrt 2020-07-13T06:10:40.326 INFO:teuthology.orchestra.run.smithi017:> sudo cp /var/lib/ceph/osd/c1-2/keyring /tmp/tmp.NTaPY6jtrt 2020-07-13T06:10:40.386 INFO:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.NTaPY6jtrt 2020-07-13T06:10:40.495 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.NTaPY6jtrt is 56B 2020-07-13T06:10:40.507 INFO:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.NTaPY6jtrt 2020-07-13T06:10:40.573 DEBUG:teuthology.orchestra.remote:smithi017:/etc/ceph/c1.client.0.keyring is 59B 2020-07-13T06:10:40.597 DEBUG:teuthology.orchestra.remote:smithi017:/etc/ceph/c1.client.1.keyring is 59B 2020-07-13T06:10:40.607 INFO:tasks.ceph:Adding keys to all mons... 2020-07-13T06:10:40.608 INFO:teuthology.orchestra.run.smithi017:> sudo tee -a /etc/ceph/c1.keyring 2020-07-13T06:10:40.624 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr.x] 2020-07-13T06:10:40.625 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDX+gtft0QvCBAARjVAz695K3QsllAk1YqCQA== 2020-07-13T06:10:40.625 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.0] 2020-07-13T06:10:40.625 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDZ+gtfESIJGBAAdUm+ALrEbfZ94mtaF/1+9w== 2020-07-13T06:10:40.625 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.1] 2020-07-13T06:10:40.625 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDb+gtfnFYnGhAALkZItF2zAJvSU12Vy3Q+eA== 2020-07-13T06:10:40.626 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.2] 2020-07-13T06:10:40.626 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDd+gtfAdX2HRAA80MAeDEz8lA0gY1reLMTMA== 2020-07-13T06:10:40.626 INFO:teuthology.orchestra.run.smithi017.stdout:[client.0] 2020-07-13T06:10:40.626 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDX+gtfvhv8DBAAc6g0bB8bWdg7PPRi9EuGdQ== 2020-07-13T06:10:40.626 INFO:teuthology.orchestra.run.smithi017.stdout:[client.1] 2020-07-13T06:10:40.627 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDX+gtfvFY/EhAAyv0M3bbCZ+tjGkfkthR4Mw== 2020-07-13T06:10:40.627 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-07-13T06:10:40.704 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-07-13T06:10:40.790 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-07-13T06:10:40.872 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-07-13T06:10:40.957 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-07-13T06:10:41.045 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-07-13T06:10:41.093 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-07-13T06:10:41.093 INFO:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2020-07-13T06:10:41.108 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2020-07-13T06:10:41.223 INFO:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2020-07-13T06:10:41.240 INFO:teuthology.orchestra.run.smithi017:> rm -- /home/ubuntu/cephtest/c1.monmap 2020-07-13T06:10:41.289 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2020-07-13T06:10:41.290 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2020-07-13T06:10:41.291 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2020-07-13T06:10:41.333 INFO:tasks.ceph.c1.mon.a:Started 2020-07-13T06:10:41.333 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2020-07-13T06:10:41.334 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2020-07-13T06:10:41.334 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2020-07-13T06:10:41.336 INFO:tasks.ceph.c1.mgr.x:Started 2020-07-13T06:10:41.337 INFO:tasks.ceph:Setting crush tunables to default 2020-07-13T06:10:41.337 INFO:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 osd crush tunables default 2020-07-13T06:10:41.913 INFO:teuthology.orchestra.run.smithi017.stderr:adjusted tunables profile to default 2020-07-13T06:10:41.924 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2020-07-13T06:10:41.924 INFO:teuthology.orchestra.run.smithi017:> mktemp 2020-07-13T06:10:41.930 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.xZSQPm2VGe 2020-07-13T06:10:41.930 INFO:teuthology.orchestra.run.smithi017:> sudo cp /var/lib/ceph/osd/c1-0/fsid /tmp/tmp.xZSQPm2VGe 2020-07-13T06:10:41.984 INFO:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.xZSQPm2VGe 2020-07-13T06:10:42.097 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.xZSQPm2VGe is 37B 2020-07-13T06:10:42.113 INFO:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.xZSQPm2VGe 2020-07-13T06:10:42.122 INFO:teuthology.orchestra.run.smithi017:> mktemp 2020-07-13T06:10:42.169 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.XSywca6WFo 2020-07-13T06:10:42.169 INFO:teuthology.orchestra.run.smithi017:> sudo cp /var/lib/ceph/osd/c1-1/fsid /tmp/tmp.XSywca6WFo 2020-07-13T06:10:42.226 INFO:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.XSywca6WFo 2020-07-13T06:10:42.330 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.XSywca6WFo is 37B 2020-07-13T06:10:42.340 INFO:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.XSywca6WFo 2020-07-13T06:10:42.346 INFO:teuthology.orchestra.run.smithi017:> mktemp 2020-07-13T06:10:42.392 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.mO9AKJa0qH 2020-07-13T06:10:42.393 INFO:teuthology.orchestra.run.smithi017:> sudo cp /var/lib/ceph/osd/c1-2/fsid /tmp/tmp.mO9AKJa0qH 2020-07-13T06:10:42.449 INFO:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.mO9AKJa0qH 2020-07-13T06:10:42.563 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.mO9AKJa0qH is 37B 2020-07-13T06:10:42.575 INFO:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.mO9AKJa0qH 2020-07-13T06:10:42.581 INFO:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 osd new df1c4b96-0c25-4ec9-889b-a9d2a3091ab5 0 2020-07-13T06:10:42.825 INFO:teuthology.orchestra.run.smithi017.stdout:0 2020-07-13T06:10:42.841 INFO:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 osd new d685cbef-0816-4051-ac44-0dbd6df16200 1 2020-07-13T06:10:43.039 INFO:teuthology.orchestra.run.smithi017.stdout:1 2020-07-13T06:10:43.051 INFO:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 osd new 9620c23e-d8db-45dc-aa87-4adfe086eee0 2 2020-07-13T06:10:43.254 INFO:teuthology.orchestra.run.smithi017.stdout:2 2020-07-13T06:10:43.270 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2020-07-13T06:10:43.270 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2020-07-13T06:10:43.272 INFO:tasks.ceph.c1.osd.0:Started 2020-07-13T06:10:43.273 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2020-07-13T06:10:43.273 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2020-07-13T06:10:43.276 INFO:tasks.ceph.c1.osd.1:Started 2020-07-13T06:10:43.276 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2020-07-13T06:10:43.277 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2020-07-13T06:10:43.278 INFO:tasks.ceph.c1.osd.2:Started 2020-07-13T06:10:43.279 INFO:tasks.ceph:Waiting for OSDs to come up 2020-07-13T06:10:43.587 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-07-13T06:10:43.765 INFO:teuthology.misc.health.smithi017.stdout: 2020-07-13T06:10:43.766 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":5,"fsid":"a3ab2581-a560-49c3-bc72-913c25ea7119","created":"2020-07-13T06:10:41.446991+0000","modified":"2020-07-13T06:10:43.253373+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"df1c4b96-0c25-4ec9-889b-a9d2a3091ab5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d685cbef-0816-4051-ac44-0dbd6df16200","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9620c23e-d8db-45dc-aa87-4adfe086eee0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-13T06:10:43.776 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-07-13T06:10:43.881 INFO:tasks.ceph.c1.osd.2.smithi017.stderr:2020-07-13T06:10:43.879+0000 7fb488a4bd80 -1 Falling back to public interface 2020-07-13T06:10:43.882 INFO:tasks.ceph.c1.osd.1.smithi017.stderr:2020-07-13T06:10:43.879+0000 7fc007c2fd80 -1 Falling back to public interface 2020-07-13T06:10:43.897 INFO:tasks.ceph.c1.osd.0.smithi017.stderr:2020-07-13T06:10:43.895+0000 7fe2d858dd80 -1 Falling back to public interface 2020-07-13T06:10:44.510 INFO:tasks.ceph.c1.osd.1.smithi017.stderr:2020-07-13T06:10:44.507+0000 7fc007c2fd80 -1 osd.1 0 log_to_monitors {default=true} 2020-07-13T06:10:44.514 INFO:tasks.ceph.c1.osd.2.smithi017.stderr:2020-07-13T06:10:44.511+0000 7fb488a4bd80 -1 osd.2 0 log_to_monitors {default=true} 2020-07-13T06:10:44.550 INFO:tasks.ceph.c1.osd.0.smithi017.stderr:2020-07-13T06:10:44.547+0000 7fe2d858dd80 -1 osd.0 0 log_to_monitors {default=true} 2020-07-13T06:10:46.476 INFO:tasks.ceph.c1.osd.0.smithi017.stderr:2020-07-13T06:10:46.471+0000 7fe2cfa58700 -1 osd.0 0 waiting for initial osdmap 2020-07-13T06:10:46.476 INFO:tasks.ceph.c1.osd.2.smithi017.stderr:2020-07-13T06:10:46.471+0000 7fb47ff16700 -1 osd.2 0 waiting for initial osdmap 2020-07-13T06:10:46.477 INFO:tasks.ceph.c1.osd.1.smithi017.stderr:2020-07-13T06:10:46.471+0000 7fbfff0fa700 -1 osd.1 0 waiting for initial osdmap 2020-07-13T06:10:50.080 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-07-13T06:10:50.343 INFO:teuthology.misc.health.smithi017.stdout: 2020-07-13T06:10:50.343 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"a3ab2581-a560-49c3-bc72-913c25ea7119","created":"2020-07-13T06:10:41.446991+0000","modified":"2020-07-13T06:10:49.548475+0000","last_up_change":"2020-07-13T06:10:47.468887+0000","last_in_change":"2020-07-13T06:10:47.468887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-13T06:10:45.575427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"df1c4b96-0c25-4ec9-889b-a9d2a3091ab5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":11400},{"type":"v1","addr":"172.21.15.17:6817","nonce":11400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":11400},{"type":"v1","addr":"172.21.15.17:6819","nonce":11400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":11400},{"type":"v1","addr":"172.21.15.17:6823","nonce":11400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":11400},{"type":"v1","addr":"172.21.15.17:6821","nonce":11400}]},"public_addr":"172.21.15.17:6817/11400","cluster_addr":"172.21.15.17:6819/11400","heartbeat_back_addr":"172.21.15.17:6823/11400","heartbeat_front_addr":"172.21.15.17:6821/11400","state":["exists","up"]},{"osd":1,"uuid":"d685cbef-0816-4051-ac44-0dbd6df16200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6803","nonce":11401},{"type":"v1","addr":"172.21.15.17:6805","nonce":11401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6807","nonce":11401},{"type":"v1","addr":"172.21.15.17:6809","nonce":11401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":11401},{"type":"v1","addr":"172.21.15.17:6815","nonce":11401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6811","nonce":11401},{"type":"v1","addr":"172.21.15.17:6813","nonce":11401}]},"public_addr":"172.21.15.17:6805/11401","cluster_addr":"172.21.15.17:6809/11401","heartbeat_back_addr":"172.21.15.17:6815/11401","heartbeat_front_addr":"172.21.15.17:6813/11401","state":["exists","up"]},{"osd":2,"uuid":"9620c23e-d8db-45dc-aa87-4adfe086eee0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":11402},{"type":"v1","addr":"172.21.15.17:6801","nonce":11402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":11402},{"type":"v1","addr":"172.21.15.17:6804","nonce":11402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":11402},{"type":"v1","addr":"172.21.15.17:6812","nonce":11402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":11402},{"type":"v1","addr":"172.21.15.17:6808","nonce":11402}]},"public_addr":"172.21.15.17:6801/11402","cluster_addr":"172.21.15.17:6804/11402","heartbeat_back_addr":"172.21.15.17:6812/11402","heartbeat_front_addr":"172.21.15.17:6808/11402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:10:45.562471+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:10:45.484930+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:10:45.473870+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-13T06:10:50.355 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-07-13T06:10:50.356 INFO:tasks.ceph:Creating RBD pool 2020-07-13T06:10:50.356 INFO:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 osd pool create rbd 8 2020-07-13T06:10:51.537 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'rbd' created 2020-07-13T06:10:51.549 INFO:teuthology.orchestra.run.smithi017:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-07-13T06:10:52.546 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rbd' on pool 'rbd' 2020-07-13T06:10:52.561 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2020-07-13T06:10:52.562 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-07-13T06:10:52.563 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-07-13T06:10:52.832 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:52.832 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":12,"fsid":"a3ab2581-a560-49c3-bc72-913c25ea7119","created":"2020-07-13T06:10:41.446991+0000","modified":"2020-07-13T06:10:52.524314+0000","last_up_change":"2020-07-13T06:10:47.468887+0000","last_in_change":"2020-07-13T06:10:47.468887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-13T06:10:45.575427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-07-13T06:10:50.631519+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"df1c4b96-0c25-4ec9-889b-a9d2a3091ab5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":11400},{"type":"v1","addr":"172.21.15.17:6817","nonce":11400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":11400},{"type":"v1","addr":"172.21.15.17:6819","nonce":11400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":11400},{"type":"v1","addr":"172.21.15.17:6823","nonce":11400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":11400},{"type":"v1","addr":"172.21.15.17:6821","nonce":11400}]},"public_addr":"172.21.15.17:6817/11400","cluster_addr":"172.21.15.17:6819/11400","heartbeat_back_addr":"172.21.15.17:6823/11400","heartbeat_front_addr":"172.21.15.17:6821/11400","state":["exists","up"]},{"osd":1,"uuid":"d685cbef-0816-4051-ac44-0dbd6df16200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6803","nonce":11401},{"type":"v1","addr":"172.21.15.17:6805","nonce":11401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6807","nonce":11401},{"type":"v1","addr":"172.21.15.17:6809","nonce":11401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":11401},{"type":"v1","addr":"172.21.15.17:6815","nonce":11401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6811","nonce":11401},{"type":"v1","addr":"172.21.15.17:6813","nonce":11401}]},"public_addr":"172.21.15.17:6805/11401","cluster_addr":"172.21.15.17:6809/11401","heartbeat_back_addr":"172.21.15.17:6815/11401","heartbeat_front_addr":"172.21.15.17:6813/11401","state":["exists","up"]},{"osd":2,"uuid":"9620c23e-d8db-45dc-aa87-4adfe086eee0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":11402},{"type":"v1","addr":"172.21.15.17:6801","nonce":11402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":11402},{"type":"v1","addr":"172.21.15.17:6804","nonce":11402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":11402},{"type":"v1","addr":"172.21.15.17:6812","nonce":11402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":11402},{"type":"v1","addr":"172.21.15.17:6808","nonce":11402}]},"public_addr":"172.21.15.17:6801/11402","cluster_addr":"172.21.15.17:6804/11402","heartbeat_back_addr":"172.21.15.17:6812/11402","heartbeat_front_addr":"172.21.15.17:6808/11402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:10:45.562471+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:10:45.484930+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:10:45.473870+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-13T06:10:52.845 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-07-13T06:10:45.575427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-07-13T06:10:50.631519+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-07-13T06:10:52.846 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get device_health_metrics pg_num 2020-07-13T06:10:53.115 INFO:teuthology.orchestra.run.smithi017.stdout:pg_num: 1 2020-07-13T06:10:53.128 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get rbd pg_num 2020-07-13T06:10:53.398 INFO:teuthology.orchestra.run.smithi017.stdout:pg_num: 8 2020-07-13T06:10:53.412 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2020-07-13T06:10:53.412 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2020-07-13T06:10:53.413 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early mgr dump --format=json 2020-07-13T06:10:53.722 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:53.735 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":11199},{"type":"v1","addr":"172.21.15.17:6825","nonce":11199}]},"active_addr":"172.21.15.17:6825/11199","active_change":"2020-07-13T06:10:45.530439+0000","active_mgr_features":4540138292837744639,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":1448866792}]},{"addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":1068957148}]},{"addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":3219521722}]}]}} 2020-07-13T06:10:53.738 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2020-07-13T06:10:53.738 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2020-07-13T06:10:53.739 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-07-13T06:10:54.003 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:54.003 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":13,"fsid":"a3ab2581-a560-49c3-bc72-913c25ea7119","created":"2020-07-13T06:10:41.446991+0000","modified":"2020-07-13T06:10:53.530231+0000","last_up_change":"2020-07-13T06:10:47.468887+0000","last_in_change":"2020-07-13T06:10:47.468887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-13T06:10:45.575427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-07-13T06:10:50.631519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"df1c4b96-0c25-4ec9-889b-a9d2a3091ab5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":11400},{"type":"v1","addr":"172.21.15.17:6817","nonce":11400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":11400},{"type":"v1","addr":"172.21.15.17:6819","nonce":11400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":11400},{"type":"v1","addr":"172.21.15.17:6823","nonce":11400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":11400},{"type":"v1","addr":"172.21.15.17:6821","nonce":11400}]},"public_addr":"172.21.15.17:6817/11400","cluster_addr":"172.21.15.17:6819/11400","heartbeat_back_addr":"172.21.15.17:6823/11400","heartbeat_front_addr":"172.21.15.17:6821/11400","state":["exists","up"]},{"osd":1,"uuid":"d685cbef-0816-4051-ac44-0dbd6df16200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6803","nonce":11401},{"type":"v1","addr":"172.21.15.17:6805","nonce":11401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6807","nonce":11401},{"type":"v1","addr":"172.21.15.17:6809","nonce":11401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":11401},{"type":"v1","addr":"172.21.15.17:6815","nonce":11401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6811","nonce":11401},{"type":"v1","addr":"172.21.15.17:6813","nonce":11401}]},"public_addr":"172.21.15.17:6805/11401","cluster_addr":"172.21.15.17:6809/11401","heartbeat_back_addr":"172.21.15.17:6815/11401","heartbeat_front_addr":"172.21.15.17:6813/11401","state":["exists","up"]},{"osd":2,"uuid":"9620c23e-d8db-45dc-aa87-4adfe086eee0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":11402},{"type":"v1","addr":"172.21.15.17:6801","nonce":11402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":11402},{"type":"v1","addr":"172.21.15.17:6804","nonce":11402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":11402},{"type":"v1","addr":"172.21.15.17:6812","nonce":11402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":11402},{"type":"v1","addr":"172.21.15.17:6808","nonce":11402}]},"public_addr":"172.21.15.17:6801/11402","cluster_addr":"172.21.15.17:6804/11402","heartbeat_back_addr":"172.21.15.17:6812/11402","heartbeat_front_addr":"172.21.15.17:6808/11402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:10:45.562471+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:10:45.484930+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:10:45.473870+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-13T06:10:54.024 INFO:tasks.ceph.ceph_manager.c1:all up! 2020-07-13T06:10:54.024 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-07-13T06:10:54.290 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:54.291 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":13,"fsid":"a3ab2581-a560-49c3-bc72-913c25ea7119","created":"2020-07-13T06:10:41.446991+0000","modified":"2020-07-13T06:10:53.530231+0000","last_up_change":"2020-07-13T06:10:47.468887+0000","last_in_change":"2020-07-13T06:10:47.468887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-13T06:10:45.575427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-07-13T06:10:50.631519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"df1c4b96-0c25-4ec9-889b-a9d2a3091ab5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":11400},{"type":"v1","addr":"172.21.15.17:6817","nonce":11400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":11400},{"type":"v1","addr":"172.21.15.17:6819","nonce":11400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":11400},{"type":"v1","addr":"172.21.15.17:6823","nonce":11400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":11400},{"type":"v1","addr":"172.21.15.17:6821","nonce":11400}]},"public_addr":"172.21.15.17:6817/11400","cluster_addr":"172.21.15.17:6819/11400","heartbeat_back_addr":"172.21.15.17:6823/11400","heartbeat_front_addr":"172.21.15.17:6821/11400","state":["exists","up"]},{"osd":1,"uuid":"d685cbef-0816-4051-ac44-0dbd6df16200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6803","nonce":11401},{"type":"v1","addr":"172.21.15.17:6805","nonce":11401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6807","nonce":11401},{"type":"v1","addr":"172.21.15.17:6809","nonce":11401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":11401},{"type":"v1","addr":"172.21.15.17:6815","nonce":11401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6811","nonce":11401},{"type":"v1","addr":"172.21.15.17:6813","nonce":11401}]},"public_addr":"172.21.15.17:6805/11401","cluster_addr":"172.21.15.17:6809/11401","heartbeat_back_addr":"172.21.15.17:6815/11401","heartbeat_front_addr":"172.21.15.17:6813/11401","state":["exists","up"]},{"osd":2,"uuid":"9620c23e-d8db-45dc-aa87-4adfe086eee0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":11402},{"type":"v1","addr":"172.21.15.17:6801","nonce":11402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":11402},{"type":"v1","addr":"172.21.15.17:6804","nonce":11402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":11402},{"type":"v1","addr":"172.21.15.17:6812","nonce":11402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":11402},{"type":"v1","addr":"172.21.15.17:6808","nonce":11402}]},"public_addr":"172.21.15.17:6801/11402","cluster_addr":"172.21.15.17:6804/11402","heartbeat_back_addr":"172.21.15.17:6812/11402","heartbeat_front_addr":"172.21.15.17:6808/11402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:10:45.562471+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:10:45.484930+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:10:45.473870+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-13T06:10:54.303 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.0 flush_pg_stats 2020-07-13T06:10:54.462 INFO:teuthology.orchestra.run.smithi017.stdout:34359738372 2020-07-13T06:10:54.463 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.1 flush_pg_stats 2020-07-13T06:10:54.628 INFO:teuthology.orchestra.run.smithi017.stdout:34359738372 2020-07-13T06:10:54.629 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.2 flush_pg_stats 2020-07-13T06:10:54.788 INFO:teuthology.orchestra.run.smithi017.stdout:34359738372 2020-07-13T06:10:54.789 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-07-13T06:10:55.072 INFO:teuthology.orchestra.run.smithi017.stdout:34359738371 2020-07-13T06:10:55.086 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.0 2020-07-13T06:10:56.087 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-07-13T06:10:56.370 INFO:teuthology.orchestra.run.smithi017.stdout:34359738372 2020-07-13T06:10:56.389 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.0 2020-07-13T06:10:56.389 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.1 2020-07-13T06:10:56.660 INFO:teuthology.orchestra.run.smithi017.stdout:34359738372 2020-07-13T06:10:56.673 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.1 2020-07-13T06:10:56.674 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.2 2020-07-13T06:10:56.943 INFO:teuthology.orchestra.run.smithi017.stdout:34359738372 2020-07-13T06:10:56.956 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.2 2020-07-13T06:10:56.957 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2020-07-13T06:10:56.957 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2020-07-13T06:10:57.219 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:57.221 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2020-07-13T06:10:57.241 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2020-07-13T06:10:55.540437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146080,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426720,"statfs":{"total":322122547200,"available":318900961280,"internally_reserved":3221225472,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.008142"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:10:52.548589+0000","last_change":"2020-07-13T06:10:52.547584+0000","last_active":"2020-07-13T06:10:52.548589+0000","last_peered":"2020-07-13T06:10:52.548589+0000","last_clean":"2020-07-13T06:10:52.548589+0000","last_became_active":"2020-07-13T06:10:52.546829+0000","last_became_peered":"2020-07-13T06:10:52.546829+0000","last_unstale":"2020-07-13T06:10:52.548589+0000","last_undegraded":"2020-07-13T06:10:52.548589+0000","last_fullsized":"2020-07-13T06:10:52.548589+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_clean_scrub_stamp":"2020-07-13T06:10:51.514373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:10:52.548574+0000","last_change":"2020-07-13T06:10:52.547646+0000","last_active":"2020-07-13T06:10:52.548574+0000","last_peered":"2020-07-13T06:10:52.548574+0000","last_clean":"2020-07-13T06:10:52.548574+0000","last_became_active":"2020-07-13T06:10:52.547063+0000","last_became_peered":"2020-07-13T06:10:52.547063+0000","last_unstale":"2020-07-13T06:10:52.548574+0000","last_undegraded":"2020-07-13T06:10:52.548574+0000","last_fullsized":"2020-07-13T06:10:52.548574+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_clean_scrub_stamp":"2020-07-13T06:10:51.514373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:10:52.548535+0000","last_change":"2020-07-13T06:10:52.547405+0000","last_active":"2020-07-13T06:10:52.548535+0000","last_peered":"2020-07-13T06:10:52.548535+0000","last_clean":"2020-07-13T06:10:52.548535+0000","last_became_active":"2020-07-13T06:10:52.546739+0000","last_became_peered":"2020-07-13T06:10:52.546739+0000","last_unstale":"2020-07-13T06:10:52.548535+0000","last_undegraded":"2020-07-13T06:10:52.548535+0000","last_fullsized":"2020-07-13T06:10:52.548535+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_clean_scrub_stamp":"2020-07-13T06:10:51.514373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:10:52.546708+0000","last_change":"2020-07-13T06:10:52.545492+0000","last_active":"2020-07-13T06:10:52.546708+0000","last_peered":"2020-07-13T06:10:52.546708+0000","last_clean":"2020-07-13T06:10:52.546708+0000","last_became_active":"2020-07-13T06:10:52.544861+0000","last_became_peered":"2020-07-13T06:10:52.544861+0000","last_unstale":"2020-07-13T06:10:52.546708+0000","last_undegraded":"2020-07-13T06:10:52.546708+0000","last_fullsized":"2020-07-13T06:10:52.546708+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_clean_scrub_stamp":"2020-07-13T06:10:51.514373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:10:52.541083+0000","last_change":"2020-07-13T06:10:49.222308+0000","last_active":"2020-07-13T06:10:52.541083+0000","last_peered":"2020-07-13T06:10:52.541083+0000","last_clean":"2020-07-13T06:10:52.541083+0000","last_became_active":"2020-07-13T06:10:49.221079+0000","last_became_peered":"2020-07-13T06:10:49.221079+0000","last_unstale":"2020-07-13T06:10:52.541083+0000","last_undegraded":"2020-07-13T06:10:52.541083+0000","last_fullsized":"2020-07-13T06:10:52.541083+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:10:46.462082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:10:46.462082+0000","last_clean_scrub_stamp":"2020-07-13T06:10:46.462082+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:10:52.546261+0000","last_change":"2020-07-13T06:10:52.545672+0000","last_active":"2020-07-13T06:10:52.546261+0000","last_peered":"2020-07-13T06:10:52.546261+0000","last_clean":"2020-07-13T06:10:52.546261+0000","last_became_active":"2020-07-13T06:10:52.545325+0000","last_became_peered":"2020-07-13T06:10:52.545325+0000","last_unstale":"2020-07-13T06:10:52.546261+0000","last_undegraded":"2020-07-13T06:10:52.546261+0000","last_fullsized":"2020-07-13T06:10:52.546261+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_clean_scrub_stamp":"2020-07-13T06:10:51.514373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:10:52.546662+0000","last_change":"2020-07-13T06:10:52.544719+0000","last_active":"2020-07-13T06:10:52.546662+0000","last_peered":"2020-07-13T06:10:52.546662+0000","last_clean":"2020-07-13T06:10:52.546662+0000","last_became_active":"2020-07-13T06:10:52.544269+0000","last_became_peered":"2020-07-13T06:10:52.544269+0000","last_unstale":"2020-07-13T06:10:52.546662+0000","last_undegraded":"2020-07-13T06:10:52.546662+0000","last_fullsized":"2020-07-13T06:10:52.546662+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_clean_scrub_stamp":"2020-07-13T06:10:51.514373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:10:52.549925+0000","last_change":"2020-07-13T06:10:52.549152+0000","last_active":"2020-07-13T06:10:52.549925+0000","last_peered":"2020-07-13T06:10:52.549925+0000","last_clean":"2020-07-13T06:10:52.549925+0000","last_became_active":"2020-07-13T06:10:52.548338+0000","last_became_peered":"2020-07-13T06:10:52.548338+0000","last_unstale":"2020-07-13T06:10:52.549925+0000","last_undegraded":"2020-07-13T06:10:52.549925+0000","last_fullsized":"2020-07-13T06:10:52.549925+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_clean_scrub_stamp":"2020-07-13T06:10:51.514373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:10:52.548405+0000","last_change":"2020-07-13T06:10:52.547328+0000","last_active":"2020-07-13T06:10:52.548405+0000","last_peered":"2020-07-13T06:10:52.548405+0000","last_clean":"2020-07-13T06:10:52.548405+0000","last_became_active":"2020-07-13T06:10:52.546683+0000","last_became_peered":"2020-07-13T06:10:52.546683+0000","last_unstale":"2020-07-13T06:10:52.548405+0000","last_undegraded":"2020-07-13T06:10:52.548405+0000","last_fullsized":"2020-07-13T06:10:52.548405+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_clean_scrub_stamp":"2020-07-13T06:10:51.514373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.482}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.369}]}]}]}} 2020-07-13T06:10:57.243 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2020-07-13T06:10:57.515 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:57.517 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2020-07-13T06:10:57.528 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2020-07-13T06:10:55.540437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146080,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426720,"statfs":{"total":322122547200,"available":318900961280,"internally_reserved":3221225472,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.008142"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:10:52.548589+0000","last_change":"2020-07-13T06:10:52.547584+0000","last_active":"2020-07-13T06:10:52.548589+0000","last_peered":"2020-07-13T06:10:52.548589+0000","last_clean":"2020-07-13T06:10:52.548589+0000","last_became_active":"2020-07-13T06:10:52.546829+0000","last_became_peered":"2020-07-13T06:10:52.546829+0000","last_unstale":"2020-07-13T06:10:52.548589+0000","last_undegraded":"2020-07-13T06:10:52.548589+0000","last_fullsized":"2020-07-13T06:10:52.548589+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_clean_scrub_stamp":"2020-07-13T06:10:51.514373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:10:52.548574+0000","last_change":"2020-07-13T06:10:52.547646+0000","last_active":"2020-07-13T06:10:52.548574+0000","last_peered":"2020-07-13T06:10:52.548574+0000","last_clean":"2020-07-13T06:10:52.548574+0000","last_became_active":"2020-07-13T06:10:52.547063+0000","last_became_peered":"2020-07-13T06:10:52.547063+0000","last_unstale":"2020-07-13T06:10:52.548574+0000","last_undegraded":"2020-07-13T06:10:52.548574+0000","last_fullsized":"2020-07-13T06:10:52.548574+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_clean_scrub_stamp":"2020-07-13T06:10:51.514373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:10:52.548535+0000","last_change":"2020-07-13T06:10:52.547405+0000","last_active":"2020-07-13T06:10:52.548535+0000","last_peered":"2020-07-13T06:10:52.548535+0000","last_clean":"2020-07-13T06:10:52.548535+0000","last_became_active":"2020-07-13T06:10:52.546739+0000","last_became_peered":"2020-07-13T06:10:52.546739+0000","last_unstale":"2020-07-13T06:10:52.548535+0000","last_undegraded":"2020-07-13T06:10:52.548535+0000","last_fullsized":"2020-07-13T06:10:52.548535+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_clean_scrub_stamp":"2020-07-13T06:10:51.514373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:10:52.546708+0000","last_change":"2020-07-13T06:10:52.545492+0000","last_active":"2020-07-13T06:10:52.546708+0000","last_peered":"2020-07-13T06:10:52.546708+0000","last_clean":"2020-07-13T06:10:52.546708+0000","last_became_active":"2020-07-13T06:10:52.544861+0000","last_became_peered":"2020-07-13T06:10:52.544861+0000","last_unstale":"2020-07-13T06:10:52.546708+0000","last_undegraded":"2020-07-13T06:10:52.546708+0000","last_fullsized":"2020-07-13T06:10:52.546708+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_clean_scrub_stamp":"2020-07-13T06:10:51.514373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:10:52.541083+0000","last_change":"2020-07-13T06:10:49.222308+0000","last_active":"2020-07-13T06:10:52.541083+0000","last_peered":"2020-07-13T06:10:52.541083+0000","last_clean":"2020-07-13T06:10:52.541083+0000","last_became_active":"2020-07-13T06:10:49.221079+0000","last_became_peered":"2020-07-13T06:10:49.221079+0000","last_unstale":"2020-07-13T06:10:52.541083+0000","last_undegraded":"2020-07-13T06:10:52.541083+0000","last_fullsized":"2020-07-13T06:10:52.541083+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:10:46.462082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:10:46.462082+0000","last_clean_scrub_stamp":"2020-07-13T06:10:46.462082+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:10:52.546261+0000","last_change":"2020-07-13T06:10:52.545672+0000","last_active":"2020-07-13T06:10:52.546261+0000","last_peered":"2020-07-13T06:10:52.546261+0000","last_clean":"2020-07-13T06:10:52.546261+0000","last_became_active":"2020-07-13T06:10:52.545325+0000","last_became_peered":"2020-07-13T06:10:52.545325+0000","last_unstale":"2020-07-13T06:10:52.546261+0000","last_undegraded":"2020-07-13T06:10:52.546261+0000","last_fullsized":"2020-07-13T06:10:52.546261+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_clean_scrub_stamp":"2020-07-13T06:10:51.514373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:10:52.546662+0000","last_change":"2020-07-13T06:10:52.544719+0000","last_active":"2020-07-13T06:10:52.546662+0000","last_peered":"2020-07-13T06:10:52.546662+0000","last_clean":"2020-07-13T06:10:52.546662+0000","last_became_active":"2020-07-13T06:10:52.544269+0000","last_became_peered":"2020-07-13T06:10:52.544269+0000","last_unstale":"2020-07-13T06:10:52.546662+0000","last_undegraded":"2020-07-13T06:10:52.546662+0000","last_fullsized":"2020-07-13T06:10:52.546662+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_clean_scrub_stamp":"2020-07-13T06:10:51.514373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:10:52.549925+0000","last_change":"2020-07-13T06:10:52.549152+0000","last_active":"2020-07-13T06:10:52.549925+0000","last_peered":"2020-07-13T06:10:52.549925+0000","last_clean":"2020-07-13T06:10:52.549925+0000","last_became_active":"2020-07-13T06:10:52.548338+0000","last_became_peered":"2020-07-13T06:10:52.548338+0000","last_unstale":"2020-07-13T06:10:52.549925+0000","last_undegraded":"2020-07-13T06:10:52.549925+0000","last_fullsized":"2020-07-13T06:10:52.549925+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_clean_scrub_stamp":"2020-07-13T06:10:51.514373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:10:52.548405+0000","last_change":"2020-07-13T06:10:52.547328+0000","last_active":"2020-07-13T06:10:52.548405+0000","last_peered":"2020-07-13T06:10:52.548405+0000","last_clean":"2020-07-13T06:10:52.548405+0000","last_became_active":"2020-07-13T06:10:52.546683+0000","last_became_peered":"2020-07-13T06:10:52.546683+0000","last_unstale":"2020-07-13T06:10:52.548405+0000","last_undegraded":"2020-07-13T06:10:52.548405+0000","last_fullsized":"2020-07-13T06:10:52.548405+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:10:51.514373+0000","last_clean_scrub_stamp":"2020-07-13T06:10:51.514373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.482}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.369}]}]}]}} 2020-07-13T06:10:57.529 INFO:tasks.ceph.ceph_manager.c1:clean! 2020-07-13T06:10:57.530 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2020-07-13T06:10:57.530 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2020-07-13T06:10:57.531 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early health --format=json 2020-07-13T06:10:57.847 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:57.867 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-07-13T06:10:57.867 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2020-07-13T06:10:57.868 INFO:teuthology.run_tasks:Running task ceph... 2020-07-13T06:10:57.879 INFO:tasks.ceph:Creating ceph cluster c2... 2020-07-13T06:10:57.879 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-07-13T06:10:57.880 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2020-07-13_03:05:02-rgw-master-distro-basic-smithi/5218593', 'branch': 'master', 'description': 'rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '5218593', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2020-07-13_03:05:02-rgw-master-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'master'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '1ac5c74502da6dc8ad965625df452eb42a846489', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '1ac5c74502da6dc8ad965625df452eb42a846489'}}, 'rgw': {'compression type': 'random', 'frontend': 'civetweb'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'master', 'sha1': '1ac5c74502da6dc8ad965625df452eb42a846489'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 1000, 'repo': 'git://git.ceph.com/ceph.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '1ac5c74502da6dc8ad965625df452eb42a846489', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'master', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_master/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph.git', 'suite_sha1': '1ac5c74502da6dc8ad965625df452eb42a846489', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '1ac5c74502da6dc8ad965625df452eb42a846489', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '1ac5c74502da6dc8ad965625df452eb42a846489', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.20773', 'targets': {'smithi017.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5Tg3xFvlmPJANVeH97ibsyYtT0l3duWosTCQdsiAKn2wAntdiIolQt5T5sNUSlPRj+sUH7eK5AF/+lijwzESp3GGzDRFrFoaWRbWOZeCGjTUUIKFFhlrf3hcXZCVQj9+Fz/1mUiZiM6dRnNEvvrFt7485R3OGrVpU6/QWvggV8Mqv/bLIerFbS1Z+3pD8jIFuepUKMV5IaHyXQrzmWESGmyR+cJYGu2CmjNEovrX3jzF9xCx+wFysKyorm0WzbveD4dQ+OyLixMa7xyO8/1i7nolBBmFltVOcYNuKeXTdRRZr+Hna3U/RutVzRuSPcOaXB6q39zmAeMC+6OIOuBz9', 'smithi014.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmQcdwKZF703ribUpzgoiJtFPNwO6YJpt3QdUhjrgrYsyMrNdTc6ePfhsPyQiUS7ch6f83BQanIliWLoIfxdHNHrOSdPtLtj569MND1/HJAnCfRm7Abc26MHr63Ej2ZrA3yfLTNz4p6na1Gm4wo0N3jBFuIP9kuXZeHoeTjIuU01e8Hcqmacb3ptSYlAaoqkblDgblem+kz6eF1yXiNCEydEgT2GKZ9djV/w8zYRrsWBu1b6J+Rq2Tn/Jp1XYFupOMo4Bpi6Ztm8GSlc0sMo0nETDec3Mwgpf/wcG65bYFVHkPPIvPl2u6VZ5yE0qaO+zPFz409WyM4gRPRkZV5lPL'}} 2020-07-13T06:10:57.881 INFO:teuthology.orchestra.run.smithi014:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-07-13T06:10:57.884 INFO:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-07-13T06:10:57.891 INFO:teuthology.orchestra.run.smithi014:> sudo install -d -m0777 -- /var/run/ceph 2020-07-13T06:10:57.932 INFO:teuthology.orchestra.run.smithi017:> sudo install -d -m0777 -- /var/run/ceph 2020-07-13T06:10:57.994 DEBUG:teuthology.orchestra.remote:smithi014:/scratch_devs is 72B 2020-07-13T06:10:58.057 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-07-13T06:10:58.058 INFO:teuthology.orchestra.run.smithi014:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-07-13T06:10:58.070 INFO:teuthology.orchestra.run.smithi014.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-07-13T06:10:58.070 INFO:teuthology.orchestra.run.smithi014.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-13T06:10:58.071 INFO:teuthology.orchestra.run.smithi014.stdout:Device: 6h/6d Inode: 343 Links: 1 2020-07-13T06:10:58.071 INFO:teuthology.orchestra.run.smithi014.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-13T06:10:58.071 INFO:teuthology.orchestra.run.smithi014.stdout:Access: 2020-07-13 06:08:03.645932511 +0000 2020-07-13T06:10:58.071 INFO:teuthology.orchestra.run.smithi014.stdout:Modify: 2020-07-13 06:08:03.525932513 +0000 2020-07-13T06:10:58.071 INFO:teuthology.orchestra.run.smithi014.stdout:Change: 2020-07-13 06:08:03.525932513 +0000 2020-07-13T06:10:58.072 INFO:teuthology.orchestra.run.smithi014.stdout: Birth: - 2020-07-13T06:10:58.078 INFO:teuthology.orchestra.run.smithi014.stderr:1+0 records in 2020-07-13T06:10:58.079 INFO:teuthology.orchestra.run.smithi014.stderr:1+0 records out 2020-07-13T06:10:58.079 INFO:teuthology.orchestra.run.smithi014.stderr:512 bytes copied, 0.000291782 s, 1.8 MB/s 2020-07-13T06:10:58.083 INFO:teuthology.orchestra.run.smithi014:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-07-13T06:10:58.133 INFO:teuthology.orchestra.run.smithi014.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-07-13T06:10:58.133 INFO:teuthology.orchestra.run.smithi014.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-13T06:10:58.133 INFO:teuthology.orchestra.run.smithi014.stdout:Device: 6h/6d Inode: 350 Links: 1 2020-07-13T06:10:58.134 INFO:teuthology.orchestra.run.smithi014.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-13T06:10:58.134 INFO:teuthology.orchestra.run.smithi014.stdout:Access: 2020-07-13 06:08:03.941932504 +0000 2020-07-13T06:10:58.134 INFO:teuthology.orchestra.run.smithi014.stdout:Modify: 2020-07-13 06:08:03.785932508 +0000 2020-07-13T06:10:58.134 INFO:teuthology.orchestra.run.smithi014.stdout:Change: 2020-07-13 06:08:03.785932508 +0000 2020-07-13T06:10:58.134 INFO:teuthology.orchestra.run.smithi014.stdout: Birth: - 2020-07-13T06:10:58.141 INFO:teuthology.orchestra.run.smithi014.stderr:1+0 records in 2020-07-13T06:10:58.142 INFO:teuthology.orchestra.run.smithi014.stderr:1+0 records out 2020-07-13T06:10:58.142 INFO:teuthology.orchestra.run.smithi014.stderr:512 bytes copied, 0.000360794 s, 1.4 MB/s 2020-07-13T06:10:58.145 INFO:teuthology.orchestra.run.smithi014:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-07-13T06:10:58.193 INFO:teuthology.orchestra.run.smithi014.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-07-13T06:10:58.193 INFO:teuthology.orchestra.run.smithi014.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-13T06:10:58.193 INFO:teuthology.orchestra.run.smithi014.stdout:Device: 6h/6d Inode: 356 Links: 1 2020-07-13T06:10:58.194 INFO:teuthology.orchestra.run.smithi014.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-13T06:10:58.194 INFO:teuthology.orchestra.run.smithi014.stdout:Access: 2020-07-13 06:08:04.201932498 +0000 2020-07-13T06:10:58.194 INFO:teuthology.orchestra.run.smithi014.stdout:Modify: 2020-07-13 06:08:04.073932501 +0000 2020-07-13T06:10:58.194 INFO:teuthology.orchestra.run.smithi014.stdout:Change: 2020-07-13 06:08:04.073932501 +0000 2020-07-13T06:10:58.194 INFO:teuthology.orchestra.run.smithi014.stdout: Birth: - 2020-07-13T06:10:58.207 INFO:teuthology.orchestra.run.smithi014.stderr:1+0 records in 2020-07-13T06:10:58.208 INFO:teuthology.orchestra.run.smithi014.stderr:1+0 records out 2020-07-13T06:10:58.208 INFO:teuthology.orchestra.run.smithi014.stderr:512 bytes copied, 0.000405443 s, 1.3 MB/s 2020-07-13T06:10:58.208 INFO:teuthology.orchestra.run.smithi014:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-07-13T06:10:58.257 INFO:teuthology.orchestra.run.smithi014.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-07-13T06:10:58.258 INFO:teuthology.orchestra.run.smithi014.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-13T06:10:58.258 INFO:teuthology.orchestra.run.smithi014.stdout:Device: 6h/6d Inode: 362 Links: 1 2020-07-13T06:10:58.259 INFO:teuthology.orchestra.run.smithi014.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-13T06:10:58.259 INFO:teuthology.orchestra.run.smithi014.stdout:Access: 2020-07-13 06:08:04.457932493 +0000 2020-07-13T06:10:58.259 INFO:teuthology.orchestra.run.smithi014.stdout:Modify: 2020-07-13 06:08:04.317932496 +0000 2020-07-13T06:10:58.259 INFO:teuthology.orchestra.run.smithi014.stdout:Change: 2020-07-13 06:08:04.317932496 +0000 2020-07-13T06:10:58.259 INFO:teuthology.orchestra.run.smithi014.stdout: Birth: - 2020-07-13T06:10:58.267 INFO:teuthology.orchestra.run.smithi014.stderr:1+0 records in 2020-07-13T06:10:58.267 INFO:teuthology.orchestra.run.smithi014.stderr:1+0 records out 2020-07-13T06:10:58.267 INFO:teuthology.orchestra.run.smithi014.stderr:512 bytes copied, 0.000374427 s, 1.4 MB/s 2020-07-13T06:10:58.271 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-13T06:10:58.272 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi014.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-07-13T06:10:58.272 INFO:tasks.ceph:Generating config... 2020-07-13T06:10:58.274 INFO:tasks.ceph:[client] debug rgw = 20 2020-07-13T06:10:58.274 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-07-13T06:10:58.274 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-07-13T06:10:58.275 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-07-13T06:10:58.275 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-07-13T06:10:58.275 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-07-13T06:10:58.275 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-07-13T06:10:58.276 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-07-13T06:10:58.276 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-07-13T06:10:58.276 INFO:tasks.ceph:[client] setgroup = ceph 2020-07-13T06:10:58.277 INFO:tasks.ceph:[client] setuser = ceph 2020-07-13T06:10:58.277 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-07-13T06:10:58.277 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-07-13T06:10:58.277 INFO:tasks.ceph:[mgr] debug ms = 1 2020-07-13T06:10:58.278 INFO:tasks.ceph:[mon] debug mon = 20 2020-07-13T06:10:58.278 INFO:tasks.ceph:[mon] debug ms = 1 2020-07-13T06:10:58.278 INFO:tasks.ceph:[mon] debug paxos = 20 2020-07-13T06:10:58.279 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-07-13T06:10:58.279 INFO:tasks.ceph:[osd] debug filestore = 20 2020-07-13T06:10:58.279 INFO:tasks.ceph:[osd] debug journal = 20 2020-07-13T06:10:58.280 INFO:tasks.ceph:[osd] debug ms = 20 2020-07-13T06:10:58.280 INFO:tasks.ceph:[osd] debug osd = 25 2020-07-13T06:10:58.280 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-07-13T06:10:58.281 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-07-13T06:10:58.281 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-07-13T06:10:58.281 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-07-13T06:10:58.282 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-07-13T06:10:58.282 INFO:tasks.ceph:Setting up c2.mon.a... 2020-07-13T06:10:58.283 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2020-07-13T06:10:58.346 INFO:teuthology.orchestra.run.smithi014.stdout:creating /etc/ceph/c2.keyring 2020-07-13T06:10:58.348 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2020-07-13T06:10:58.431 INFO:teuthology.orchestra.run.smithi014:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-07-13T06:10:58.489 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.14')] 2020-07-13T06:10:58.490 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.14 --print /home/ubuntu/cephtest/c2.monmap 2020-07-13T06:10:58.558 INFO:teuthology.orchestra.run.smithi014.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2020-07-13T06:10:58.558 INFO:teuthology.orchestra.run.smithi014.stdout:monmaptool: generated fsid 40c31169-cbb7-4010-a96c-b2a8967a70ad 2020-07-13T06:10:58.558 INFO:teuthology.orchestra.run.smithi014.stdout:epoch 0 2020-07-13T06:10:58.559 INFO:teuthology.orchestra.run.smithi014.stdout:fsid 40c31169-cbb7-4010-a96c-b2a8967a70ad 2020-07-13T06:10:58.559 INFO:teuthology.orchestra.run.smithi014.stdout:last_changed 2020-07-13T06:10:58.560871+0000 2020-07-13T06:10:58.560 INFO:teuthology.orchestra.run.smithi014.stdout:created 2020-07-13T06:10:58.560871+0000 2020-07-13T06:10:58.560 INFO:teuthology.orchestra.run.smithi014.stdout:min_mon_release 0 (unknown) 2020-07-13T06:10:58.560 INFO:teuthology.orchestra.run.smithi014.stdout:0: [v2:172.21.15.14:3300/0,v1:172.21.15.14:6789/0] mon.a 2020-07-13T06:10:58.560 INFO:teuthology.orchestra.run.smithi014.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2020-07-13T06:10:58.561 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 40c31169-cbb7-4010-a96c-b2a8967a70ad... 2020-07-13T06:10:58.562 INFO:teuthology.orchestra.run.smithi014:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-07-13T06:10:58.604 INFO:teuthology.orchestra.run.smithi017:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-07-13T06:10:58.635 INFO:teuthology.orchestra.run.smithi014.stdout:[global] 2020-07-13T06:10:58.635 INFO:teuthology.orchestra.run.smithi014.stdout: chdir = "" 2020-07-13T06:10:58.635 INFO:teuthology.orchestra.run.smithi014.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-07-13T06:10:58.636 INFO:teuthology.orchestra.run.smithi014.stdout: auth supported = cephx 2020-07-13T06:10:58.636 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.636 INFO:teuthology.orchestra.run.smithi014.stdout: filestore xattr use omap = true 2020-07-13T06:10:58.636 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.636 INFO:teuthology.orchestra.run.smithi014.stdout: mon clock drift allowed = 1.000 2020-07-13T06:10:58.637 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.637 INFO:teuthology.orchestra.run.smithi014.stdout: osd crush chooseleaf type = 0 2020-07-13T06:10:58.637 INFO:teuthology.orchestra.run.smithi014.stdout: auth debug = true 2020-07-13T06:10:58.637 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.637 INFO:teuthology.orchestra.run.smithi014.stdout: ms die on old message = true 2020-07-13T06:10:58.638 INFO:teuthology.orchestra.run.smithi014.stdout: ms die on bug = true 2020-07-13T06:10:58.638 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.638 INFO:teuthology.orchestra.run.smithi014.stdout: mon max pg per osd = 10000 # >= luminous 2020-07-13T06:10:58.638 INFO:teuthology.orchestra.run.smithi014.stdout: mon pg warn max object skew = 0 2020-07-13T06:10:58.638 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.639 INFO:teuthology.orchestra.run.smithi014.stdout: # disable pg_autoscaler by default for new pools 2020-07-13T06:10:58.639 INFO:teuthology.orchestra.run.smithi014.stdout: osd_pool_default_pg_autoscale_mode = off 2020-07-13T06:10:58.639 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.639 INFO:teuthology.orchestra.run.smithi014.stdout: osd pool default size = 2 2020-07-13T06:10:58.639 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.640 INFO:teuthology.orchestra.run.smithi014.stdout: mon osd allow primary affinity = true 2020-07-13T06:10:58.640 INFO:teuthology.orchestra.run.smithi014.stdout: mon osd allow pg remap = true 2020-07-13T06:10:58.640 INFO:teuthology.orchestra.run.smithi014.stdout: mon warn on legacy crush tunables = false 2020-07-13T06:10:58.640 INFO:teuthology.orchestra.run.smithi014.stdout: mon warn on crush straw calc version zero = false 2020-07-13T06:10:58.640 INFO:teuthology.orchestra.run.smithi014.stdout: mon warn on no sortbitwise = false 2020-07-13T06:10:58.640 INFO:teuthology.orchestra.run.smithi014.stdout: mon warn on osd down out interval zero = false 2020-07-13T06:10:58.641 INFO:teuthology.orchestra.run.smithi014.stdout: mon warn on too few osds = false 2020-07-13T06:10:58.641 INFO:teuthology.orchestra.run.smithi014.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-07-13T06:10:58.641 INFO:teuthology.orchestra.run.smithi014.stdout: mon_warn_on_pool_no_redundancy = false 2020-07-13T06:10:58.641 INFO:teuthology.orchestra.run.smithi014.stdout: mon_allow_pool_size_one = true 2020-07-13T06:10:58.641 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.642 INFO:teuthology.orchestra.run.smithi014.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-07-13T06:10:58.642 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.642 INFO:teuthology.orchestra.run.smithi014.stdout: osd default data pool replay window = 5 2020-07-13T06:10:58.642 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.643 INFO:teuthology.orchestra.run.smithi014.stdout: mon allow pool delete = true 2020-07-13T06:10:58.643 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.643 INFO:teuthology.orchestra.run.smithi014.stdout: mon cluster log file level = debug 2020-07-13T06:10:58.643 INFO:teuthology.orchestra.run.smithi014.stdout: debug asserts on shutdown = true 2020-07-13T06:10:58.644 INFO:teuthology.orchestra.run.smithi014.stdout: mon host = 172.21.15.14 2020-07-13T06:10:58.644 INFO:teuthology.orchestra.run.smithi014.stdout: osd heartbeat grace = 40 2020-07-13T06:10:58.644 INFO:teuthology.orchestra.run.smithi014.stdout: fsid = 40c31169-cbb7-4010-a96c-b2a8967a70ad 2020-07-13T06:10:58.644 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.644 INFO:teuthology.orchestra.run.smithi014.stdout:[osd] 2020-07-13T06:10:58.645 INFO:teuthology.orchestra.run.smithi014.stdout: osd journal size = 100 2020-07-13T06:10:58.645 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.645 INFO:teuthology.orchestra.run.smithi014.stdout: osd scrub load threshold = 5.0 2020-07-13T06:10:58.645 INFO:teuthology.orchestra.run.smithi014.stdout: osd scrub max interval = 600 2020-07-13T06:10:58.645 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.646 INFO:teuthology.orchestra.run.smithi014.stdout: osd recover clone overlap = true 2020-07-13T06:10:58.646 INFO:teuthology.orchestra.run.smithi014.stdout: osd recovery max chunk = 1048576 2020-07-13T06:10:58.646 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.646 INFO:teuthology.orchestra.run.smithi014.stdout: osd debug shutdown = true 2020-07-13T06:10:58.646 INFO:teuthology.orchestra.run.smithi014.stdout: osd debug op order = true 2020-07-13T06:10:58.647 INFO:teuthology.orchestra.run.smithi014.stdout: osd debug verify stray on activate = true 2020-07-13T06:10:58.647 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.647 INFO:teuthology.orchestra.run.smithi014.stdout: osd open classes on start = true 2020-07-13T06:10:58.647 INFO:teuthology.orchestra.run.smithi014.stdout: osd debug pg log writeout = true 2020-07-13T06:10:58.647 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.648 INFO:teuthology.orchestra.run.smithi014.stdout: osd deep scrub update digest min age = 30 2020-07-13T06:10:58.648 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.648 INFO:teuthology.orchestra.run.smithi014.stdout: osd map max advance = 10 2020-07-13T06:10:58.648 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.649 INFO:teuthology.orchestra.run.smithi014.stdout: journal zero on create = true 2020-07-13T06:10:58.649 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.649 INFO:teuthology.orchestra.run.smithi014.stdout: filestore ondisk finisher threads = 3 2020-07-13T06:10:58.649 INFO:teuthology.orchestra.run.smithi014.stdout: filestore apply finisher threads = 3 2020-07-13T06:10:58.649 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.650 INFO:teuthology.orchestra.run.smithi014.stdout: bdev debug aio = true 2020-07-13T06:10:58.650 INFO:teuthology.orchestra.run.smithi014.stdout: osd debug misdirected ops = true 2020-07-13T06:10:58.650 INFO:teuthology.orchestra.run.smithi014.stdout: debug filestore = 20 2020-07-13T06:10:58.650 INFO:teuthology.orchestra.run.smithi014.stdout: debug journal = 20 2020-07-13T06:10:58.651 INFO:teuthology.orchestra.run.smithi014.stdout: debug ms = 20 2020-07-13T06:10:58.651 INFO:teuthology.orchestra.run.smithi014.stdout: debug osd = 25 2020-07-13T06:10:58.651 INFO:teuthology.orchestra.run.smithi014.stdout: osd fast shutdown = False 2020-07-13T06:10:58.651 INFO:teuthology.orchestra.run.smithi014.stdout: rocksdb delete range threshold = 0 2020-07-13T06:10:58.651 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.652 INFO:teuthology.orchestra.run.smithi014.stdout:[mgr] 2020-07-13T06:10:58.652 INFO:teuthology.orchestra.run.smithi014.stdout: debug ms = 1 2020-07-13T06:10:58.652 INFO:teuthology.orchestra.run.smithi014.stdout: debug mgr = 20 2020-07-13T06:10:58.652 INFO:teuthology.orchestra.run.smithi014.stdout: debug mon = 20 2020-07-13T06:10:58.653 INFO:teuthology.orchestra.run.smithi014.stdout: debug auth = 20 2020-07-13T06:10:58.653 INFO:teuthology.orchestra.run.smithi014.stdout: mon reweight min pgs per osd = 4 2020-07-13T06:10:58.653 INFO:teuthology.orchestra.run.smithi014.stdout: mon reweight min bytes per osd = 10 2020-07-13T06:10:58.653 INFO:teuthology.orchestra.run.smithi014.stdout: mgr/telemetry/nag = false 2020-07-13T06:10:58.653 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.654 INFO:teuthology.orchestra.run.smithi014.stdout:[mon] 2020-07-13T06:10:58.654 INFO:teuthology.orchestra.run.smithi014.stdout: debug ms = 1 2020-07-13T06:10:58.655 INFO:teuthology.orchestra.run.smithi014.stdout: debug mon = 20 2020-07-13T06:10:58.655 INFO:teuthology.orchestra.run.smithi014.stdout: debug paxos = 20 2020-07-13T06:10:58.656 INFO:teuthology.orchestra.run.smithi014.stdout: debug auth = 20 2020-07-13T06:10:58.656 INFO:teuthology.orchestra.run.smithi014.stdout: mon data avail warn = 5 2020-07-13T06:10:58.656 INFO:teuthology.orchestra.run.smithi014.stdout: mon mgr mkfs grace = 240 2020-07-13T06:10:58.656 INFO:teuthology.orchestra.run.smithi014.stdout: mon reweight min pgs per osd = 4 2020-07-13T06:10:58.656 INFO:teuthology.orchestra.run.smithi014.stdout: mon osd reporter subtree level = osd 2020-07-13T06:10:58.657 INFO:teuthology.orchestra.run.smithi014.stdout: mon osd prime pg temp = true 2020-07-13T06:10:58.657 INFO:teuthology.orchestra.run.smithi014.stdout: mon reweight min bytes per osd = 10 2020-07-13T06:10:58.657 INFO:teuthology.orchestra.run.smithi014.stdout: mon osd crush smoke test = False 2020-07-13T06:10:58.657 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:10:58.657 INFO:teuthology.orchestra.run.smithi014.stdout:[client] 2020-07-13T06:10:58.658 INFO:teuthology.orchestra.run.smithi014.stdout: rgw cache enabled = true 2020-07-13T06:10:58.658 INFO:teuthology.orchestra.run.smithi014.stdout: rgw enable ops log = true 2020-07-13T06:10:58.658 INFO:teuthology.orchestra.run.smithi014.stdout: rgw enable usage log = true 2020-07-13T06:10:58.658 INFO:teuthology.orchestra.run.smithi014.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-07-13T06:10:58.659 INFO:teuthology.orchestra.run.smithi014.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-07-13T06:10:58.659 INFO:teuthology.orchestra.run.smithi014.stdout: debug rgw = 20 2020-07-13T06:10:58.659 INFO:teuthology.orchestra.run.smithi014.stdout: rgw crypt require ssl = False 2020-07-13T06:10:58.660 INFO:teuthology.orchestra.run.smithi014.stdout: rgw crypt s3 kms backend = testing 2020-07-13T06:10:58.660 INFO:teuthology.orchestra.run.smithi014.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-07-13T06:10:58.660 INFO:teuthology.orchestra.run.smithi014.stdout: rgw curl low speed time = 300 2020-07-13T06:10:58.660 INFO:teuthology.orchestra.run.smithi014.stdout: rgw data log num shards = 4 2020-07-13T06:10:58.661 INFO:teuthology.orchestra.run.smithi014.stdout: rgw md log max shards = 4 2020-07-13T06:10:58.661 INFO:teuthology.orchestra.run.smithi014.stdout: rgw sync log trim interval = 0 2020-07-13T06:10:58.661 INFO:teuthology.orchestra.run.smithi014.stdout: rgw sync obj etag verify = True 2020-07-13T06:10:58.661 INFO:teuthology.orchestra.run.smithi014.stdout: setgroup = ceph 2020-07-13T06:10:58.661 INFO:teuthology.orchestra.run.smithi014.stdout: setuser = ceph 2020-07-13T06:10:58.662 INFO:teuthology.orchestra.run.smithi014.stdout:[mon.a] 2020-07-13T06:10:58.662 INFO:teuthology.orchestra.run.smithi014.stdout:[osd.0] 2020-07-13T06:10:58.662 INFO:teuthology.orchestra.run.smithi014.stdout: osd_max_omap_entries_per_request = 10 2020-07-13T06:10:58.662 INFO:teuthology.orchestra.run.smithi014.stdout:[osd.1] 2020-07-13T06:10:58.662 INFO:teuthology.orchestra.run.smithi014.stdout: osd_max_omap_entries_per_request = 1000 2020-07-13T06:10:58.663 INFO:teuthology.orchestra.run.smithi014.stdout:[osd.2] 2020-07-13T06:10:58.663 INFO:teuthology.orchestra.run.smithi014.stdout: osd_max_omap_entries_per_request = 10000 2020-07-13T06:10:58.664 INFO:teuthology.orchestra.run.smithi017.stdout:[global] 2020-07-13T06:10:58.664 INFO:teuthology.orchestra.run.smithi017.stdout: chdir = "" 2020-07-13T06:10:58.665 INFO:teuthology.orchestra.run.smithi017.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-07-13T06:10:58.665 INFO:teuthology.orchestra.run.smithi017.stdout: auth supported = cephx 2020-07-13T06:10:58.665 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.665 INFO:teuthology.orchestra.run.smithi017.stdout: filestore xattr use omap = true 2020-07-13T06:10:58.665 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.666 INFO:teuthology.orchestra.run.smithi017.stdout: mon clock drift allowed = 1.000 2020-07-13T06:10:58.666 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.666 INFO:teuthology.orchestra.run.smithi017.stdout: osd crush chooseleaf type = 0 2020-07-13T06:10:58.666 INFO:teuthology.orchestra.run.smithi017.stdout: auth debug = true 2020-07-13T06:10:58.666 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.667 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on old message = true 2020-07-13T06:10:58.667 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on bug = true 2020-07-13T06:10:58.667 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.667 INFO:teuthology.orchestra.run.smithi017.stdout: mon max pg per osd = 10000 # >= luminous 2020-07-13T06:10:58.667 INFO:teuthology.orchestra.run.smithi017.stdout: mon pg warn max object skew = 0 2020-07-13T06:10:58.668 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.668 INFO:teuthology.orchestra.run.smithi017.stdout: # disable pg_autoscaler by default for new pools 2020-07-13T06:10:58.668 INFO:teuthology.orchestra.run.smithi017.stdout: osd_pool_default_pg_autoscale_mode = off 2020-07-13T06:10:58.668 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.669 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default size = 2 2020-07-13T06:10:58.669 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.669 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow primary affinity = true 2020-07-13T06:10:58.669 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow pg remap = true 2020-07-13T06:10:58.669 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on legacy crush tunables = false 2020-07-13T06:10:58.670 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on crush straw calc version zero = false 2020-07-13T06:10:58.670 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on no sortbitwise = false 2020-07-13T06:10:58.670 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on osd down out interval zero = false 2020-07-13T06:10:58.670 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on too few osds = false 2020-07-13T06:10:58.670 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-07-13T06:10:58.671 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_no_redundancy = false 2020-07-13T06:10:58.671 INFO:teuthology.orchestra.run.smithi017.stdout: mon_allow_pool_size_one = true 2020-07-13T06:10:58.671 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.671 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-07-13T06:10:58.671 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.672 INFO:teuthology.orchestra.run.smithi017.stdout: osd default data pool replay window = 5 2020-07-13T06:10:58.672 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.672 INFO:teuthology.orchestra.run.smithi017.stdout: mon allow pool delete = true 2020-07-13T06:10:58.673 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.673 INFO:teuthology.orchestra.run.smithi017.stdout: mon cluster log file level = debug 2020-07-13T06:10:58.673 INFO:teuthology.orchestra.run.smithi017.stdout: debug asserts on shutdown = true 2020-07-13T06:10:58.673 INFO:teuthology.orchestra.run.smithi017.stdout: mon host = 172.21.15.14 2020-07-13T06:10:58.673 INFO:teuthology.orchestra.run.smithi017.stdout: osd heartbeat grace = 40 2020-07-13T06:10:58.674 INFO:teuthology.orchestra.run.smithi017.stdout: fsid = 40c31169-cbb7-4010-a96c-b2a8967a70ad 2020-07-13T06:10:58.674 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.674 INFO:teuthology.orchestra.run.smithi017.stdout:[osd] 2020-07-13T06:10:58.674 INFO:teuthology.orchestra.run.smithi017.stdout: osd journal size = 100 2020-07-13T06:10:58.674 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.674 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub load threshold = 5.0 2020-07-13T06:10:58.675 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub max interval = 600 2020-07-13T06:10:58.675 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.675 INFO:teuthology.orchestra.run.smithi017.stdout: osd recover clone overlap = true 2020-07-13T06:10:58.675 INFO:teuthology.orchestra.run.smithi017.stdout: osd recovery max chunk = 1048576 2020-07-13T06:10:58.675 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.676 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug shutdown = true 2020-07-13T06:10:58.676 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug op order = true 2020-07-13T06:10:58.676 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug verify stray on activate = true 2020-07-13T06:10:58.676 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.676 INFO:teuthology.orchestra.run.smithi017.stdout: osd open classes on start = true 2020-07-13T06:10:58.676 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug pg log writeout = true 2020-07-13T06:10:58.677 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.677 INFO:teuthology.orchestra.run.smithi017.stdout: osd deep scrub update digest min age = 30 2020-07-13T06:10:58.677 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.677 INFO:teuthology.orchestra.run.smithi017.stdout: osd map max advance = 10 2020-07-13T06:10:58.680 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.681 INFO:teuthology.orchestra.run.smithi017.stdout: journal zero on create = true 2020-07-13T06:10:58.681 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.681 INFO:teuthology.orchestra.run.smithi017.stdout: filestore ondisk finisher threads = 3 2020-07-13T06:10:58.681 INFO:teuthology.orchestra.run.smithi017.stdout: filestore apply finisher threads = 3 2020-07-13T06:10:58.682 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.682 INFO:teuthology.orchestra.run.smithi017.stdout: bdev debug aio = true 2020-07-13T06:10:58.682 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug misdirected ops = true 2020-07-13T06:10:58.682 INFO:teuthology.orchestra.run.smithi017.stdout: debug filestore = 20 2020-07-13T06:10:58.683 INFO:teuthology.orchestra.run.smithi017.stdout: debug journal = 20 2020-07-13T06:10:58.683 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 20 2020-07-13T06:10:58.683 INFO:teuthology.orchestra.run.smithi017.stdout: debug osd = 25 2020-07-13T06:10:58.683 INFO:teuthology.orchestra.run.smithi017.stdout: osd fast shutdown = False 2020-07-13T06:10:58.683 INFO:teuthology.orchestra.run.smithi017.stdout: rocksdb delete range threshold = 0 2020-07-13T06:10:58.684 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.684 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr] 2020-07-13T06:10:58.684 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2020-07-13T06:10:58.684 INFO:teuthology.orchestra.run.smithi017.stdout: debug mgr = 20 2020-07-13T06:10:58.684 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2020-07-13T06:10:58.685 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2020-07-13T06:10:58.685 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2020-07-13T06:10:58.685 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2020-07-13T06:10:58.686 INFO:teuthology.orchestra.run.smithi017.stdout: mgr/telemetry/nag = false 2020-07-13T06:10:58.686 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.686 INFO:teuthology.orchestra.run.smithi017.stdout:[mon] 2020-07-13T06:10:58.686 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2020-07-13T06:10:58.686 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2020-07-13T06:10:58.686 INFO:teuthology.orchestra.run.smithi017.stdout: debug paxos = 20 2020-07-13T06:10:58.687 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2020-07-13T06:10:58.687 INFO:teuthology.orchestra.run.smithi017.stdout: mon data avail warn = 5 2020-07-13T06:10:58.687 INFO:teuthology.orchestra.run.smithi017.stdout: mon mgr mkfs grace = 240 2020-07-13T06:10:58.687 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2020-07-13T06:10:58.687 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd reporter subtree level = osd 2020-07-13T06:10:58.688 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd prime pg temp = true 2020-07-13T06:10:58.688 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2020-07-13T06:10:58.688 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd crush smoke test = False 2020-07-13T06:10:58.688 INFO:teuthology.orchestra.run.smithi017.stdout: 2020-07-13T06:10:58.688 INFO:teuthology.orchestra.run.smithi017.stdout:[client] 2020-07-13T06:10:58.689 INFO:teuthology.orchestra.run.smithi017.stdout: rgw cache enabled = true 2020-07-13T06:10:58.689 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable ops log = true 2020-07-13T06:10:58.689 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable usage log = true 2020-07-13T06:10:58.689 INFO:teuthology.orchestra.run.smithi017.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-07-13T06:10:58.689 INFO:teuthology.orchestra.run.smithi017.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-07-13T06:10:58.689 INFO:teuthology.orchestra.run.smithi017.stdout: debug rgw = 20 2020-07-13T06:10:58.690 INFO:teuthology.orchestra.run.smithi017.stdout: rgw crypt require ssl = False 2020-07-13T06:10:58.690 INFO:teuthology.orchestra.run.smithi017.stdout: rgw crypt s3 kms backend = testing 2020-07-13T06:10:58.690 INFO:teuthology.orchestra.run.smithi017.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-07-13T06:10:58.690 INFO:teuthology.orchestra.run.smithi017.stdout: rgw curl low speed time = 300 2020-07-13T06:10:58.690 INFO:teuthology.orchestra.run.smithi017.stdout: rgw data log num shards = 4 2020-07-13T06:10:58.691 INFO:teuthology.orchestra.run.smithi017.stdout: rgw md log max shards = 4 2020-07-13T06:10:58.691 INFO:teuthology.orchestra.run.smithi017.stdout: rgw sync log trim interval = 0 2020-07-13T06:10:58.691 INFO:teuthology.orchestra.run.smithi017.stdout: rgw sync obj etag verify = True 2020-07-13T06:10:58.691 INFO:teuthology.orchestra.run.smithi017.stdout: setgroup = ceph 2020-07-13T06:10:58.691 INFO:teuthology.orchestra.run.smithi017.stdout: setuser = ceph 2020-07-13T06:10:58.691 INFO:teuthology.orchestra.run.smithi017.stdout:[mon.a] 2020-07-13T06:10:58.692 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.0] 2020-07-13T06:10:58.692 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_omap_entries_per_request = 10 2020-07-13T06:10:58.692 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.1] 2020-07-13T06:10:58.692 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_omap_entries_per_request = 1000 2020-07-13T06:10:58.692 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.2] 2020-07-13T06:10:58.693 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_omap_entries_per_request = 10000 2020-07-13T06:10:58.693 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2020-07-13T06:10:58.694 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2020-07-13T06:10:58.753 INFO:tasks.ceph:Copying monmap to all nodes... 2020-07-13T06:10:58.762 DEBUG:teuthology.orchestra.remote:smithi014:/etc/ceph/c2.keyring is 206B 2020-07-13T06:10:58.780 DEBUG:teuthology.orchestra.remote:smithi014:/home/ubuntu/cephtest/c2.monmap is 179B 2020-07-13T06:10:58.789 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi017.front.sepia.ceph.com 2020-07-13T06:10:58.790 INFO:teuthology.orchestra.run.smithi017:> sudo sh -c 'cat > /etc/ceph/c2.keyring' && sudo chmod 0644 /etc/ceph/c2.keyring 2020-07-13T06:10:58.847 INFO:teuthology.orchestra.run.smithi017:> cat > /home/ubuntu/cephtest/c2.monmap 2020-07-13T06:10:58.936 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi014.front.sepia.ceph.com 2020-07-13T06:10:58.937 INFO:teuthology.orchestra.run.smithi014:> sudo sh -c 'cat > /etc/ceph/c2.keyring' && sudo chmod 0644 /etc/ceph/c2.keyring 2020-07-13T06:10:58.963 INFO:teuthology.orchestra.run.smithi014:> cat > /home/ubuntu/cephtest/c2.monmap 2020-07-13T06:10:59.052 INFO:tasks.ceph:Setting up mon nodes... 2020-07-13T06:10:59.052 INFO:tasks.ceph:Setting up mgr nodes... 2020-07-13T06:10:59.053 INFO:teuthology.orchestra.run.smithi014:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2020-07-13T06:10:59.135 INFO:teuthology.orchestra.run.smithi014.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2020-07-13T06:10:59.138 INFO:tasks.ceph:Setting up mds nodes... 2020-07-13T06:10:59.138 INFO:tasks.ceph_client:Setting up client nodes... 2020-07-13T06:10:59.139 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2020-07-13T06:10:59.211 INFO:teuthology.orchestra.run.smithi014.stdout:creating /etc/ceph/c2.client.0.keyring 2020-07-13T06:10:59.222 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2020-07-13T06:10:59.295 INFO:teuthology.orchestra.run.smithi014.stdout:creating /etc/ceph/c2.client.1.keyring 2020-07-13T06:10:59.306 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-07-13T06:10:59.306 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi017.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi014.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-07-13T06:10:59.307 INFO:teuthology.orchestra.run.smithi014:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2020-07-13T06:10:59.362 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-13T06:10:59.363 INFO:tasks.ceph:role: c2.osd.0 2020-07-13T06:10:59.364 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi014.front.sepia.ceph.com 2020-07-13T06:10:59.365 INFO:teuthology.orchestra.run.smithi014:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-07-13T06:10:59.874 INFO:teuthology.orchestra.run.smithi014.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-07-13T06:10:59.875 INFO:teuthology.orchestra.run.smithi014.stdout: = sectsz=512 attr=2, projid32bit=1 2020-07-13T06:10:59.875 INFO:teuthology.orchestra.run.smithi014.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-07-13T06:10:59.875 INFO:teuthology.orchestra.run.smithi014.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-07-13T06:10:59.875 INFO:teuthology.orchestra.run.smithi014.stdout: = sunit=0 swidth=0 blks 2020-07-13T06:10:59.876 INFO:teuthology.orchestra.run.smithi014.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-07-13T06:10:59.876 INFO:teuthology.orchestra.run.smithi014.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-07-13T06:10:59.876 INFO:teuthology.orchestra.run.smithi014.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-07-13T06:10:59.876 INFO:teuthology.orchestra.run.smithi014.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-07-13T06:10:59.877 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi014.front.sepia.ceph.com -o noatime 2020-07-13T06:10:59.878 INFO:teuthology.orchestra.run.smithi014:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2020-07-13T06:10:59.956 INFO:teuthology.orchestra.run.smithi014:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2020-07-13T06:10:59.969 INFO:teuthology.orchestra.run.smithi014.stderr:sudo: /sbin/restorecon: command not found 2020-07-13T06:10:59.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-07-13T06:10:59.971 INFO:teuthology.orchestra.run.smithi014:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2020-07-13T06:11:00.027 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-13T06:11:00.027 INFO:tasks.ceph:role: c2.osd.1 2020-07-13T06:11:00.028 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi014.front.sepia.ceph.com 2020-07-13T06:11:00.028 INFO:teuthology.orchestra.run.smithi014:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-07-13T06:11:00.554 INFO:teuthology.orchestra.run.smithi014.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-07-13T06:11:00.555 INFO:teuthology.orchestra.run.smithi014.stdout: = sectsz=512 attr=2, projid32bit=1 2020-07-13T06:11:00.555 INFO:teuthology.orchestra.run.smithi014.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-07-13T06:11:00.555 INFO:teuthology.orchestra.run.smithi014.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-07-13T06:11:00.555 INFO:teuthology.orchestra.run.smithi014.stdout: = sunit=0 swidth=0 blks 2020-07-13T06:11:00.556 INFO:teuthology.orchestra.run.smithi014.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-07-13T06:11:00.556 INFO:teuthology.orchestra.run.smithi014.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-07-13T06:11:00.556 INFO:teuthology.orchestra.run.smithi014.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-07-13T06:11:00.556 INFO:teuthology.orchestra.run.smithi014.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-07-13T06:11:00.557 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi014.front.sepia.ceph.com -o noatime 2020-07-13T06:11:00.558 INFO:teuthology.orchestra.run.smithi014:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2020-07-13T06:11:00.607 INFO:teuthology.orchestra.run.smithi014:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2020-07-13T06:11:00.624 INFO:teuthology.orchestra.run.smithi014.stderr:sudo: /sbin/restorecon: command not found 2020-07-13T06:11:00.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-07-13T06:11:00.625 INFO:teuthology.orchestra.run.smithi014:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2020-07-13T06:11:00.686 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-13T06:11:00.686 INFO:tasks.ceph:role: c2.osd.2 2020-07-13T06:11:00.687 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi014.front.sepia.ceph.com 2020-07-13T06:11:00.687 INFO:teuthology.orchestra.run.smithi014:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-07-13T06:11:01.186 INFO:teuthology.orchestra.run.smithi014.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-07-13T06:11:01.187 INFO:teuthology.orchestra.run.smithi014.stdout: = sectsz=512 attr=2, projid32bit=1 2020-07-13T06:11:01.187 INFO:teuthology.orchestra.run.smithi014.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-07-13T06:11:01.187 INFO:teuthology.orchestra.run.smithi014.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-07-13T06:11:01.187 INFO:teuthology.orchestra.run.smithi014.stdout: = sunit=0 swidth=0 blks 2020-07-13T06:11:01.187 INFO:teuthology.orchestra.run.smithi014.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-07-13T06:11:01.188 INFO:teuthology.orchestra.run.smithi014.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-07-13T06:11:01.188 INFO:teuthology.orchestra.run.smithi014.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-07-13T06:11:01.188 INFO:teuthology.orchestra.run.smithi014.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-07-13T06:11:01.189 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi014.front.sepia.ceph.com -o noatime 2020-07-13T06:11:01.189 INFO:teuthology.orchestra.run.smithi014:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2020-07-13T06:11:01.236 INFO:teuthology.orchestra.run.smithi014:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2020-07-13T06:11:01.246 INFO:teuthology.orchestra.run.smithi014.stderr:sudo: /sbin/restorecon: command not found 2020-07-13T06:11:01.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-07-13T06:11:01.248 INFO:teuthology.orchestra.run.smithi014:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2020-07-13T06:11:01.364 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:01.357+0000 7f63b5608d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2020-07-13T06:11:01.365 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:01.361+0000 7f63b5608d80 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2020-07-13T06:11:01.365 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:01.361+0000 7f63b5608d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-07-13T06:11:01.365 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:01.361+0000 7f63b5608d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-07-13T06:11:01.366 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:01.361+0000 7f63b5608d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-07-13T06:11:01.366 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:01.365+0000 7f63b5608d80 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2020-07-13T06:11:03.314 INFO:teuthology.orchestra.run.smithi014:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2020-07-13T06:11:03.331 INFO:teuthology.orchestra.run.smithi014:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2020-07-13T06:11:03.412 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:03.413+0000 7f4f83158d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2020-07-13T06:11:03.413 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:03.413+0000 7f4f83158d80 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2020-07-13T06:11:03.413 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:03.413+0000 7f4f83158d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-07-13T06:11:03.413 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:03.413+0000 7f4f83158d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-07-13T06:11:03.414 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:03.413+0000 7f4f83158d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-07-13T06:11:03.414 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:03.413+0000 7f4f83158d80 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2020-07-13T06:11:05.358 INFO:teuthology.orchestra.run.smithi014:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2020-07-13T06:11:05.414 INFO:teuthology.orchestra.run.smithi014:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2020-07-13T06:11:05.499 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:05.501+0000 7fabd4ee2d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2020-07-13T06:11:05.499 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:05.501+0000 7fabd4ee2d80 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2020-07-13T06:11:05.500 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:05.501+0000 7fabd4ee2d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-07-13T06:11:05.500 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:05.501+0000 7fabd4ee2d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-07-13T06:11:05.500 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:05.501+0000 7fabd4ee2d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-07-13T06:11:05.500 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:05.501+0000 7fabd4ee2d80 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2020-07-13T06:11:07.429 INFO:teuthology.orchestra.run.smithi014:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2020-07-13T06:11:07.485 INFO:tasks.ceph:Reading keys from all nodes... 2020-07-13T06:11:07.486 INFO:teuthology.orchestra.run.smithi014:> mktemp 2020-07-13T06:11:07.531 INFO:teuthology.orchestra.run.smithi014.stdout:/tmp/tmp.maaeUp3Auc 2020-07-13T06:11:07.532 INFO:teuthology.orchestra.run.smithi014:> sudo cp /var/lib/ceph/mgr/c2-x/keyring /tmp/tmp.maaeUp3Auc 2020-07-13T06:11:07.589 INFO:teuthology.orchestra.run.smithi014:> sudo chmod 0666 /tmp/tmp.maaeUp3Auc 2020-07-13T06:11:07.697 DEBUG:teuthology.orchestra.remote:smithi014:/tmp/tmp.maaeUp3Auc is 56B 2020-07-13T06:11:07.708 INFO:teuthology.orchestra.run.smithi014:> rm -fr /tmp/tmp.maaeUp3Auc 2020-07-13T06:11:07.715 INFO:teuthology.orchestra.run.smithi014:> mktemp 2020-07-13T06:11:07.763 INFO:teuthology.orchestra.run.smithi014.stdout:/tmp/tmp.KY6K3BMpJf 2020-07-13T06:11:07.764 INFO:teuthology.orchestra.run.smithi014:> sudo cp /var/lib/ceph/osd/c2-0/keyring /tmp/tmp.KY6K3BMpJf 2020-07-13T06:11:07.821 INFO:teuthology.orchestra.run.smithi014:> sudo chmod 0666 /tmp/tmp.KY6K3BMpJf 2020-07-13T06:11:07.955 DEBUG:teuthology.orchestra.remote:smithi014:/tmp/tmp.KY6K3BMpJf is 56B 2020-07-13T06:11:07.966 INFO:teuthology.orchestra.run.smithi014:> rm -fr /tmp/tmp.KY6K3BMpJf 2020-07-13T06:11:07.972 INFO:teuthology.orchestra.run.smithi014:> mktemp 2020-07-13T06:11:08.019 INFO:teuthology.orchestra.run.smithi014.stdout:/tmp/tmp.RtnLL0mzEd 2020-07-13T06:11:08.020 INFO:teuthology.orchestra.run.smithi014:> sudo cp /var/lib/ceph/osd/c2-1/keyring /tmp/tmp.RtnLL0mzEd 2020-07-13T06:11:08.082 INFO:teuthology.orchestra.run.smithi014:> sudo chmod 0666 /tmp/tmp.RtnLL0mzEd 2020-07-13T06:11:08.198 DEBUG:teuthology.orchestra.remote:smithi014:/tmp/tmp.RtnLL0mzEd is 56B 2020-07-13T06:11:08.210 INFO:teuthology.orchestra.run.smithi014:> rm -fr /tmp/tmp.RtnLL0mzEd 2020-07-13T06:11:08.217 INFO:teuthology.orchestra.run.smithi014:> mktemp 2020-07-13T06:11:08.264 INFO:teuthology.orchestra.run.smithi014.stdout:/tmp/tmp.hQyRVbtUg1 2020-07-13T06:11:08.264 INFO:teuthology.orchestra.run.smithi014:> sudo cp /var/lib/ceph/osd/c2-2/keyring /tmp/tmp.hQyRVbtUg1 2020-07-13T06:11:08.329 INFO:teuthology.orchestra.run.smithi014:> sudo chmod 0666 /tmp/tmp.hQyRVbtUg1 2020-07-13T06:11:08.438 DEBUG:teuthology.orchestra.remote:smithi014:/tmp/tmp.hQyRVbtUg1 is 56B 2020-07-13T06:11:08.463 INFO:teuthology.orchestra.run.smithi014:> rm -fr /tmp/tmp.hQyRVbtUg1 2020-07-13T06:11:08.518 DEBUG:teuthology.orchestra.remote:smithi014:/etc/ceph/c2.client.0.keyring is 59B 2020-07-13T06:11:08.538 DEBUG:teuthology.orchestra.remote:smithi014:/etc/ceph/c2.client.1.keyring is 59B 2020-07-13T06:11:08.553 INFO:tasks.ceph:Adding keys to all mons... 2020-07-13T06:11:08.554 INFO:teuthology.orchestra.run.smithi014:> sudo tee -a /etc/ceph/c2.keyring 2020-07-13T06:11:08.566 INFO:teuthology.orchestra.run.smithi014.stdout:[mgr.x] 2020-07-13T06:11:08.566 INFO:teuthology.orchestra.run.smithi014.stdout: key = AQDz+gtfb2hHCBAA2zkbrxF980A6abT8UxzGBg== 2020-07-13T06:11:08.566 INFO:teuthology.orchestra.run.smithi014.stdout:[osd.0] 2020-07-13T06:11:08.567 INFO:teuthology.orchestra.run.smithi014.stdout: key = AQD1+gtft2GIFRAAXpdW5PJFBm3oVIg2lOxH5Q== 2020-07-13T06:11:08.567 INFO:teuthology.orchestra.run.smithi014.stdout:[osd.1] 2020-07-13T06:11:08.567 INFO:teuthology.orchestra.run.smithi014.stdout: key = AQD3+gtf4Dy+GBAAr1Y7dTfAx5rMKgxNhLE6iA== 2020-07-13T06:11:08.567 INFO:teuthology.orchestra.run.smithi014.stdout:[osd.2] 2020-07-13T06:11:08.568 INFO:teuthology.orchestra.run.smithi014.stdout: key = AQD5+gtfCXPrHRAAuR294oheQthsrFUCU1x4iA== 2020-07-13T06:11:08.568 INFO:teuthology.orchestra.run.smithi014.stdout:[client.0] 2020-07-13T06:11:08.568 INFO:teuthology.orchestra.run.smithi014.stdout: key = AQDz+gtfQ+LBDBAA6+3IbjvFFx77Q/YSRRc1xw== 2020-07-13T06:11:08.568 INFO:teuthology.orchestra.run.smithi014.stdout:[client.1] 2020-07-13T06:11:08.568 INFO:teuthology.orchestra.run.smithi014.stdout: key = AQDz+gtfMvrFERAA6zqOJMjrPCRHJIMCbEKCgw== 2020-07-13T06:11:08.569 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-07-13T06:11:08.654 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-07-13T06:11:08.742 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-07-13T06:11:08.829 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-07-13T06:11:08.914 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-07-13T06:11:09.009 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-07-13T06:11:09.053 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-07-13T06:11:09.053 INFO:teuthology.orchestra.run.smithi014:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2020-07-13T06:11:09.109 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2020-07-13T06:11:09.206 INFO:teuthology.orchestra.run.smithi014:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2020-07-13T06:11:09.224 INFO:teuthology.orchestra.run.smithi014:> rm -- /home/ubuntu/cephtest/c2.monmap 2020-07-13T06:11:09.273 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2020-07-13T06:11:09.274 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2020-07-13T06:11:09.274 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2020-07-13T06:11:09.316 INFO:tasks.ceph.c2.mon.a:Started 2020-07-13T06:11:09.316 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2020-07-13T06:11:09.317 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2020-07-13T06:11:09.317 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2020-07-13T06:11:09.320 INFO:tasks.ceph.c2.mgr.x:Started 2020-07-13T06:11:09.320 INFO:tasks.ceph:Setting crush tunables to default 2020-07-13T06:11:09.320 INFO:teuthology.orchestra.run.smithi014:> sudo ceph --cluster c2 osd crush tunables default 2020-07-13T06:11:09.883 INFO:teuthology.orchestra.run.smithi014.stderr:adjusted tunables profile to default 2020-07-13T06:11:09.893 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2020-07-13T06:11:09.893 INFO:teuthology.orchestra.run.smithi014:> mktemp 2020-07-13T06:11:09.897 INFO:teuthology.orchestra.run.smithi014.stdout:/tmp/tmp.2ABYeYwYpP 2020-07-13T06:11:09.898 INFO:teuthology.orchestra.run.smithi014:> sudo cp /var/lib/ceph/osd/c2-0/fsid /tmp/tmp.2ABYeYwYpP 2020-07-13T06:11:09.951 INFO:teuthology.orchestra.run.smithi014:> sudo chmod 0666 /tmp/tmp.2ABYeYwYpP 2020-07-13T06:11:10.074 DEBUG:teuthology.orchestra.remote:smithi014:/tmp/tmp.2ABYeYwYpP is 37B 2020-07-13T06:11:10.084 INFO:teuthology.orchestra.run.smithi014:> rm -fr /tmp/tmp.2ABYeYwYpP 2020-07-13T06:11:10.090 INFO:teuthology.orchestra.run.smithi014:> mktemp 2020-07-13T06:11:10.135 INFO:teuthology.orchestra.run.smithi014.stdout:/tmp/tmp.McRpg1T7oS 2020-07-13T06:11:10.136 INFO:teuthology.orchestra.run.smithi014:> sudo cp /var/lib/ceph/osd/c2-1/fsid /tmp/tmp.McRpg1T7oS 2020-07-13T06:11:10.192 INFO:teuthology.orchestra.run.smithi014:> sudo chmod 0666 /tmp/tmp.McRpg1T7oS 2020-07-13T06:11:10.310 DEBUG:teuthology.orchestra.remote:smithi014:/tmp/tmp.McRpg1T7oS is 37B 2020-07-13T06:11:10.320 INFO:teuthology.orchestra.run.smithi014:> rm -fr /tmp/tmp.McRpg1T7oS 2020-07-13T06:11:10.326 INFO:teuthology.orchestra.run.smithi014:> mktemp 2020-07-13T06:11:10.372 INFO:teuthology.orchestra.run.smithi014.stdout:/tmp/tmp.4C54Eodtkp 2020-07-13T06:11:10.372 INFO:teuthology.orchestra.run.smithi014:> sudo cp /var/lib/ceph/osd/c2-2/fsid /tmp/tmp.4C54Eodtkp 2020-07-13T06:11:10.428 INFO:teuthology.orchestra.run.smithi014:> sudo chmod 0666 /tmp/tmp.4C54Eodtkp 2020-07-13T06:11:10.544 DEBUG:teuthology.orchestra.remote:smithi014:/tmp/tmp.4C54Eodtkp is 37B 2020-07-13T06:11:10.557 INFO:teuthology.orchestra.run.smithi014:> rm -fr /tmp/tmp.4C54Eodtkp 2020-07-13T06:11:10.563 INFO:teuthology.orchestra.run.smithi014:> sudo ceph --cluster c2 osd new 13364d16-b593-4f9d-9d37-be85ee20b5f8 0 2020-07-13T06:11:10.811 INFO:teuthology.orchestra.run.smithi014.stdout:0 2020-07-13T06:11:10.822 INFO:teuthology.orchestra.run.smithi014:> sudo ceph --cluster c2 osd new b31653f9-c9bd-46ec-88b2-44ff22b034ac 1 2020-07-13T06:11:11.041 INFO:teuthology.orchestra.run.smithi014.stdout:1 2020-07-13T06:11:11.057 INFO:teuthology.orchestra.run.smithi014:> sudo ceph --cluster c2 osd new f78532d2-8618-4457-9fca-7b7c67957dbd 2 2020-07-13T06:11:11.259 INFO:teuthology.orchestra.run.smithi014.stdout:2 2020-07-13T06:11:11.271 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2020-07-13T06:11:11.272 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2020-07-13T06:11:11.274 INFO:tasks.ceph.c2.osd.0:Started 2020-07-13T06:11:11.275 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2020-07-13T06:11:11.275 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2020-07-13T06:11:11.277 INFO:tasks.ceph.c2.osd.1:Started 2020-07-13T06:11:11.278 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2020-07-13T06:11:11.278 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2020-07-13T06:11:11.280 INFO:tasks.ceph.c2.osd.2:Started 2020-07-13T06:11:11.281 INFO:tasks.ceph:Waiting for OSDs to come up 2020-07-13T06:11:11.584 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-07-13T06:11:11.764 INFO:teuthology.misc.health.smithi014.stdout: 2020-07-13T06:11:11.764 INFO:teuthology.misc.health.smithi014.stdout:{"epoch":5,"fsid":"40c31169-cbb7-4010-a96c-b2a8967a70ad","created":"2020-07-13T06:11:09.431632+0000","modified":"2020-07-13T06:11:11.259055+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"13364d16-b593-4f9d-9d37-be85ee20b5f8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b31653f9-c9bd-46ec-88b2-44ff22b034ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f78532d2-8618-4457-9fca-7b7c67957dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-13T06:11:11.774 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-07-13T06:11:11.904 INFO:tasks.ceph.c2.osd.0.smithi014.stderr:2020-07-13T06:11:11.905+0000 7fa18f412d80 -1 Falling back to public interface 2020-07-13T06:11:11.905 INFO:tasks.ceph.c2.osd.1.smithi014.stderr:2020-07-13T06:11:11.905+0000 7f2025f25d80 -1 Falling back to public interface 2020-07-13T06:11:11.921 INFO:tasks.ceph.c2.osd.2.smithi014.stderr:2020-07-13T06:11:11.921+0000 7f3f82610d80 -1 Falling back to public interface 2020-07-13T06:11:12.517 INFO:tasks.ceph.c2.osd.1.smithi014.stderr:2020-07-13T06:11:12.517+0000 7f2025f25d80 -1 osd.1 0 log_to_monitors {default=true} 2020-07-13T06:11:12.538 INFO:tasks.ceph.c2.osd.0.smithi014.stderr:2020-07-13T06:11:12.537+0000 7fa18f412d80 -1 osd.0 0 log_to_monitors {default=true} 2020-07-13T06:11:12.587 INFO:tasks.ceph.c2.osd.2.smithi014.stderr:2020-07-13T06:11:12.589+0000 7f3f82610d80 -1 osd.2 0 log_to_monitors {default=true} 2020-07-13T06:11:14.469 INFO:tasks.ceph.c2.osd.1.smithi014.stderr:2020-07-13T06:11:14.469+0000 7f201d3f0700 -1 osd.1 0 waiting for initial osdmap 2020-07-13T06:11:14.471 INFO:tasks.ceph.c2.osd.0.smithi014.stderr:2020-07-13T06:11:14.473+0000 7fa1868dd700 -1 osd.0 0 waiting for initial osdmap 2020-07-13T06:11:14.471 INFO:tasks.ceph.c2.osd.2.smithi014.stderr:2020-07-13T06:11:14.473+0000 7f3f79adb700 -1 osd.2 0 waiting for initial osdmap 2020-07-13T06:11:18.078 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-07-13T06:11:18.342 INFO:teuthology.misc.health.smithi014.stdout: 2020-07-13T06:11:18.343 INFO:teuthology.misc.health.smithi014.stdout:{"epoch":10,"fsid":"40c31169-cbb7-4010-a96c-b2a8967a70ad","created":"2020-07-13T06:11:09.431632+0000","modified":"2020-07-13T06:11:17.487859+0000","last_up_change":"2020-07-13T06:11:15.464413+0000","last_in_change":"2020-07-13T06:11:15.464413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-13T06:11:13.473547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"13364d16-b593-4f9d-9d37-be85ee20b5f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6802","nonce":11357},{"type":"v1","addr":"172.21.15.14:6805","nonce":11357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6807","nonce":11357},{"type":"v1","addr":"172.21.15.14:6809","nonce":11357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6814","nonce":11357},{"type":"v1","addr":"172.21.15.14:6815","nonce":11357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6812","nonce":11357},{"type":"v1","addr":"172.21.15.14:6813","nonce":11357}]},"public_addr":"172.21.15.14:6805/11357","cluster_addr":"172.21.15.14:6809/11357","heartbeat_back_addr":"172.21.15.14:6815/11357","heartbeat_front_addr":"172.21.15.14:6813/11357","state":["exists","up"]},{"osd":1,"uuid":"b31653f9-c9bd-46ec-88b2-44ff22b034ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6800","nonce":11358},{"type":"v1","addr":"172.21.15.14:6801","nonce":11358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6803","nonce":11358},{"type":"v1","addr":"172.21.15.14:6804","nonce":11358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6810","nonce":11358},{"type":"v1","addr":"172.21.15.14:6811","nonce":11358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6806","nonce":11358},{"type":"v1","addr":"172.21.15.14:6808","nonce":11358}]},"public_addr":"172.21.15.14:6801/11358","cluster_addr":"172.21.15.14:6804/11358","heartbeat_back_addr":"172.21.15.14:6811/11358","heartbeat_front_addr":"172.21.15.14:6808/11358","state":["exists","up"]},{"osd":2,"uuid":"f78532d2-8618-4457-9fca-7b7c67957dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6816","nonce":11375},{"type":"v1","addr":"172.21.15.14:6817","nonce":11375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6818","nonce":11375},{"type":"v1","addr":"172.21.15.14:6819","nonce":11375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6822","nonce":11375},{"type":"v1","addr":"172.21.15.14:6823","nonce":11375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6820","nonce":11375},{"type":"v1","addr":"172.21.15.14:6821","nonce":11375}]},"public_addr":"172.21.15.14:6817/11375","cluster_addr":"172.21.15.14:6819/11375","heartbeat_back_addr":"172.21.15.14:6823/11375","heartbeat_front_addr":"172.21.15.14:6821/11375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:11:13.575713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:11:13.472111+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:11:13.548223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-13T06:11:18.355 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-07-13T06:11:18.356 INFO:tasks.ceph:Creating RBD pool 2020-07-13T06:11:18.356 INFO:teuthology.orchestra.run.smithi014:> sudo ceph --cluster c2 osd pool create rbd 8 2020-07-13T06:11:19.508 INFO:teuthology.orchestra.run.smithi014.stderr:pool 'rbd' created 2020-07-13T06:11:19.525 INFO:teuthology.orchestra.run.smithi014:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-07-13T06:11:20.519 INFO:teuthology.orchestra.run.smithi014.stderr:enabled application 'rbd' on pool 'rbd' 2020-07-13T06:11:20.534 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2020-07-13T06:11:20.535 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-07-13T06:11:20.536 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-07-13T06:11:20.802 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:11:20.803 INFO:teuthology.orchestra.run.smithi014.stdout:{"epoch":12,"fsid":"40c31169-cbb7-4010-a96c-b2a8967a70ad","created":"2020-07-13T06:11:09.431632+0000","modified":"2020-07-13T06:11:20.504990+0000","last_up_change":"2020-07-13T06:11:15.464413+0000","last_in_change":"2020-07-13T06:11:15.464413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-13T06:11:13.473547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-07-13T06:11:18.633711+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"13364d16-b593-4f9d-9d37-be85ee20b5f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6802","nonce":11357},{"type":"v1","addr":"172.21.15.14:6805","nonce":11357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6807","nonce":11357},{"type":"v1","addr":"172.21.15.14:6809","nonce":11357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6814","nonce":11357},{"type":"v1","addr":"172.21.15.14:6815","nonce":11357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6812","nonce":11357},{"type":"v1","addr":"172.21.15.14:6813","nonce":11357}]},"public_addr":"172.21.15.14:6805/11357","cluster_addr":"172.21.15.14:6809/11357","heartbeat_back_addr":"172.21.15.14:6815/11357","heartbeat_front_addr":"172.21.15.14:6813/11357","state":["exists","up"]},{"osd":1,"uuid":"b31653f9-c9bd-46ec-88b2-44ff22b034ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6800","nonce":11358},{"type":"v1","addr":"172.21.15.14:6801","nonce":11358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6803","nonce":11358},{"type":"v1","addr":"172.21.15.14:6804","nonce":11358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6810","nonce":11358},{"type":"v1","addr":"172.21.15.14:6811","nonce":11358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6806","nonce":11358},{"type":"v1","addr":"172.21.15.14:6808","nonce":11358}]},"public_addr":"172.21.15.14:6801/11358","cluster_addr":"172.21.15.14:6804/11358","heartbeat_back_addr":"172.21.15.14:6811/11358","heartbeat_front_addr":"172.21.15.14:6808/11358","state":["exists","up"]},{"osd":2,"uuid":"f78532d2-8618-4457-9fca-7b7c67957dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6816","nonce":11375},{"type":"v1","addr":"172.21.15.14:6817","nonce":11375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6818","nonce":11375},{"type":"v1","addr":"172.21.15.14:6819","nonce":11375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6822","nonce":11375},{"type":"v1","addr":"172.21.15.14:6823","nonce":11375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6820","nonce":11375},{"type":"v1","addr":"172.21.15.14:6821","nonce":11375}]},"public_addr":"172.21.15.14:6817/11375","cluster_addr":"172.21.15.14:6819/11375","heartbeat_back_addr":"172.21.15.14:6823/11375","heartbeat_front_addr":"172.21.15.14:6821/11375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:11:13.575713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:11:13.472111+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:11:13.548223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-13T06:11:20.816 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-07-13T06:11:13.473547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-07-13T06:11:18.633711+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-07-13T06:11:20.816 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get device_health_metrics pg_num 2020-07-13T06:11:21.094 INFO:teuthology.orchestra.run.smithi014.stdout:pg_num: 1 2020-07-13T06:11:21.104 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get rbd pg_num 2020-07-13T06:11:21.383 INFO:teuthology.orchestra.run.smithi014.stdout:pg_num: 8 2020-07-13T06:11:21.396 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2020-07-13T06:11:21.396 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2020-07-13T06:11:21.397 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early mgr dump --format=json 2020-07-13T06:11:21.713 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:11:21.725 INFO:teuthology.orchestra.run.smithi014.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6824","nonce":11155},{"type":"v1","addr":"172.21.15.14:6825","nonce":11155}]},"active_addr":"172.21.15.14:6825/11155","active_change":"2020-07-13T06:11:13.439405+0000","active_mgr_features":4540138292837744639,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.14:0","nonce":744387446}]},{"addrvec":[{"type":"v2","addr":"172.21.15.14:0","nonce":877368796}]},{"addrvec":[{"type":"v2","addr":"172.21.15.14:0","nonce":1527255193}]}]}} 2020-07-13T06:11:21.728 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2020-07-13T06:11:21.729 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2020-07-13T06:11:21.729 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-07-13T06:11:22.007 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:11:22.008 INFO:teuthology.orchestra.run.smithi014.stdout:{"epoch":13,"fsid":"40c31169-cbb7-4010-a96c-b2a8967a70ad","created":"2020-07-13T06:11:09.431632+0000","modified":"2020-07-13T06:11:21.514343+0000","last_up_change":"2020-07-13T06:11:15.464413+0000","last_in_change":"2020-07-13T06:11:15.464413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-13T06:11:13.473547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-07-13T06:11:18.633711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"13364d16-b593-4f9d-9d37-be85ee20b5f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6802","nonce":11357},{"type":"v1","addr":"172.21.15.14:6805","nonce":11357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6807","nonce":11357},{"type":"v1","addr":"172.21.15.14:6809","nonce":11357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6814","nonce":11357},{"type":"v1","addr":"172.21.15.14:6815","nonce":11357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6812","nonce":11357},{"type":"v1","addr":"172.21.15.14:6813","nonce":11357}]},"public_addr":"172.21.15.14:6805/11357","cluster_addr":"172.21.15.14:6809/11357","heartbeat_back_addr":"172.21.15.14:6815/11357","heartbeat_front_addr":"172.21.15.14:6813/11357","state":["exists","up"]},{"osd":1,"uuid":"b31653f9-c9bd-46ec-88b2-44ff22b034ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6800","nonce":11358},{"type":"v1","addr":"172.21.15.14:6801","nonce":11358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6803","nonce":11358},{"type":"v1","addr":"172.21.15.14:6804","nonce":11358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6810","nonce":11358},{"type":"v1","addr":"172.21.15.14:6811","nonce":11358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6806","nonce":11358},{"type":"v1","addr":"172.21.15.14:6808","nonce":11358}]},"public_addr":"172.21.15.14:6801/11358","cluster_addr":"172.21.15.14:6804/11358","heartbeat_back_addr":"172.21.15.14:6811/11358","heartbeat_front_addr":"172.21.15.14:6808/11358","state":["exists","up"]},{"osd":2,"uuid":"f78532d2-8618-4457-9fca-7b7c67957dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6816","nonce":11375},{"type":"v1","addr":"172.21.15.14:6817","nonce":11375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6818","nonce":11375},{"type":"v1","addr":"172.21.15.14:6819","nonce":11375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6822","nonce":11375},{"type":"v1","addr":"172.21.15.14:6823","nonce":11375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6820","nonce":11375},{"type":"v1","addr":"172.21.15.14:6821","nonce":11375}]},"public_addr":"172.21.15.14:6817/11375","cluster_addr":"172.21.15.14:6819/11375","heartbeat_back_addr":"172.21.15.14:6823/11375","heartbeat_front_addr":"172.21.15.14:6821/11375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:11:13.575713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:11:13.472111+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:11:13.548223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-13T06:11:22.020 INFO:tasks.ceph.ceph_manager.c2:all up! 2020-07-13T06:11:22.021 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-07-13T06:11:22.293 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:11:22.293 INFO:teuthology.orchestra.run.smithi014.stdout:{"epoch":13,"fsid":"40c31169-cbb7-4010-a96c-b2a8967a70ad","created":"2020-07-13T06:11:09.431632+0000","modified":"2020-07-13T06:11:21.514343+0000","last_up_change":"2020-07-13T06:11:15.464413+0000","last_in_change":"2020-07-13T06:11:15.464413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-13T06:11:13.473547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-07-13T06:11:18.633711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"13364d16-b593-4f9d-9d37-be85ee20b5f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6802","nonce":11357},{"type":"v1","addr":"172.21.15.14:6805","nonce":11357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6807","nonce":11357},{"type":"v1","addr":"172.21.15.14:6809","nonce":11357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6814","nonce":11357},{"type":"v1","addr":"172.21.15.14:6815","nonce":11357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6812","nonce":11357},{"type":"v1","addr":"172.21.15.14:6813","nonce":11357}]},"public_addr":"172.21.15.14:6805/11357","cluster_addr":"172.21.15.14:6809/11357","heartbeat_back_addr":"172.21.15.14:6815/11357","heartbeat_front_addr":"172.21.15.14:6813/11357","state":["exists","up"]},{"osd":1,"uuid":"b31653f9-c9bd-46ec-88b2-44ff22b034ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6800","nonce":11358},{"type":"v1","addr":"172.21.15.14:6801","nonce":11358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6803","nonce":11358},{"type":"v1","addr":"172.21.15.14:6804","nonce":11358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6810","nonce":11358},{"type":"v1","addr":"172.21.15.14:6811","nonce":11358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6806","nonce":11358},{"type":"v1","addr":"172.21.15.14:6808","nonce":11358}]},"public_addr":"172.21.15.14:6801/11358","cluster_addr":"172.21.15.14:6804/11358","heartbeat_back_addr":"172.21.15.14:6811/11358","heartbeat_front_addr":"172.21.15.14:6808/11358","state":["exists","up"]},{"osd":2,"uuid":"f78532d2-8618-4457-9fca-7b7c67957dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6816","nonce":11375},{"type":"v1","addr":"172.21.15.14:6817","nonce":11375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6818","nonce":11375},{"type":"v1","addr":"172.21.15.14:6819","nonce":11375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6822","nonce":11375},{"type":"v1","addr":"172.21.15.14:6823","nonce":11375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.14:6820","nonce":11375},{"type":"v1","addr":"172.21.15.14:6821","nonce":11375}]},"public_addr":"172.21.15.14:6817/11375","cluster_addr":"172.21.15.14:6819/11375","heartbeat_back_addr":"172.21.15.14:6823/11375","heartbeat_front_addr":"172.21.15.14:6821/11375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:11:13.575713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:11:13.472111+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-13T06:11:13.548223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-13T06:11:22.306 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.0 flush_pg_stats 2020-07-13T06:11:22.471 INFO:teuthology.orchestra.run.smithi014.stdout:34359738371 2020-07-13T06:11:22.472 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.1 flush_pg_stats 2020-07-13T06:11:22.631 INFO:teuthology.orchestra.run.smithi014.stdout:34359738373 2020-07-13T06:11:22.632 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.2 flush_pg_stats 2020-07-13T06:11:22.792 INFO:teuthology.orchestra.run.smithi014.stdout:34359738371 2020-07-13T06:11:22.793 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-07-13T06:11:23.068 INFO:teuthology.orchestra.run.smithi014.stdout:34359738369 2020-07-13T06:11:23.088 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738369 for osd.0 2020-07-13T06:11:24.089 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-07-13T06:11:24.375 INFO:teuthology.orchestra.run.smithi014.stdout:34359738371 2020-07-13T06:11:24.388 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.0 2020-07-13T06:11:24.389 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.1 2020-07-13T06:11:24.660 INFO:teuthology.orchestra.run.smithi014.stdout:34359738373 2020-07-13T06:11:24.673 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738373 got 34359738373 for osd.1 2020-07-13T06:11:24.674 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.2 2020-07-13T06:11:24.947 INFO:teuthology.orchestra.run.smithi014.stdout:34359738371 2020-07-13T06:11:24.960 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.2 2020-07-13T06:11:24.960 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2020-07-13T06:11:24.961 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2020-07-13T06:11:25.230 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:11:25.232 INFO:teuthology.orchestra.run.smithi014.stderr:dumped all 2020-07-13T06:11:25.243 INFO:teuthology.orchestra.run.smithi014.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-07-13T06:11:23.449901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.931370"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:11:20.523287+0000","last_change":"2020-07-13T06:11:20.522277+0000","last_active":"2020-07-13T06:11:20.523287+0000","last_peered":"2020-07-13T06:11:20.523287+0000","last_clean":"2020-07-13T06:11:20.523287+0000","last_became_active":"2020-07-13T06:11:20.521514+0000","last_became_peered":"2020-07-13T06:11:20.521514+0000","last_unstale":"2020-07-13T06:11:20.523287+0000","last_undegraded":"2020-07-13T06:11:20.523287+0000","last_fullsized":"2020-07-13T06:11:20.523287+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_clean_scrub_stamp":"2020-07-13T06:11:19.491245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:11:20.522899+0000","last_change":"2020-07-13T06:11:20.521335+0000","last_active":"2020-07-13T06:11:20.522899+0000","last_peered":"2020-07-13T06:11:20.522899+0000","last_clean":"2020-07-13T06:11:20.522899+0000","last_became_active":"2020-07-13T06:11:20.520872+0000","last_became_peered":"2020-07-13T06:11:20.520872+0000","last_unstale":"2020-07-13T06:11:20.522899+0000","last_undegraded":"2020-07-13T06:11:20.522899+0000","last_fullsized":"2020-07-13T06:11:20.522899+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_clean_scrub_stamp":"2020-07-13T06:11:19.491245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:11:20.527873+0000","last_change":"2020-07-13T06:11:20.527270+0000","last_active":"2020-07-13T06:11:20.527873+0000","last_peered":"2020-07-13T06:11:20.527873+0000","last_clean":"2020-07-13T06:11:20.527873+0000","last_became_active":"2020-07-13T06:11:20.526889+0000","last_became_peered":"2020-07-13T06:11:20.526889+0000","last_unstale":"2020-07-13T06:11:20.527873+0000","last_undegraded":"2020-07-13T06:11:20.527873+0000","last_fullsized":"2020-07-13T06:11:20.527873+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_clean_scrub_stamp":"2020-07-13T06:11:19.491245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:11:20.524270+0000","last_change":"2020-07-13T06:11:20.524270+0000","last_active":"2020-07-13T06:11:20.524270+0000","last_peered":"2020-07-13T06:11:20.524270+0000","last_clean":"2020-07-13T06:11:20.524270+0000","last_became_active":"2020-07-13T06:11:20.523826+0000","last_became_peered":"2020-07-13T06:11:20.523826+0000","last_unstale":"2020-07-13T06:11:20.524270+0000","last_undegraded":"2020-07-13T06:11:20.524270+0000","last_fullsized":"2020-07-13T06:11:20.524270+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_clean_scrub_stamp":"2020-07-13T06:11:19.491245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:11:20.518676+0000","last_change":"2020-07-13T06:11:16.628962+0000","last_active":"2020-07-13T06:11:20.518676+0000","last_peered":"2020-07-13T06:11:20.518676+0000","last_clean":"2020-07-13T06:11:20.518676+0000","last_became_active":"2020-07-13T06:11:16.627429+0000","last_became_peered":"2020-07-13T06:11:16.627429+0000","last_unstale":"2020-07-13T06:11:20.518676+0000","last_undegraded":"2020-07-13T06:11:20.518676+0000","last_fullsized":"2020-07-13T06:11:20.518676+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:11:14.455048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:11:14.455048+0000","last_clean_scrub_stamp":"2020-07-13T06:11:14.455048+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:11:20.525639+0000","last_change":"2020-07-13T06:11:20.524616+0000","last_active":"2020-07-13T06:11:20.525639+0000","last_peered":"2020-07-13T06:11:20.525639+0000","last_clean":"2020-07-13T06:11:20.525639+0000","last_became_active":"2020-07-13T06:11:20.524129+0000","last_became_peered":"2020-07-13T06:11:20.524129+0000","last_unstale":"2020-07-13T06:11:20.525639+0000","last_undegraded":"2020-07-13T06:11:20.525639+0000","last_fullsized":"2020-07-13T06:11:20.525639+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_clean_scrub_stamp":"2020-07-13T06:11:19.491245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:11:20.523563+0000","last_change":"2020-07-13T06:11:20.523563+0000","last_active":"2020-07-13T06:11:20.523563+0000","last_peered":"2020-07-13T06:11:20.523563+0000","last_clean":"2020-07-13T06:11:20.523563+0000","last_became_active":"2020-07-13T06:11:20.522980+0000","last_became_peered":"2020-07-13T06:11:20.522980+0000","last_unstale":"2020-07-13T06:11:20.523563+0000","last_undegraded":"2020-07-13T06:11:20.523563+0000","last_fullsized":"2020-07-13T06:11:20.523563+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_clean_scrub_stamp":"2020-07-13T06:11:19.491245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:11:20.525944+0000","last_change":"2020-07-13T06:11:20.525944+0000","last_active":"2020-07-13T06:11:20.525944+0000","last_peered":"2020-07-13T06:11:20.525944+0000","last_clean":"2020-07-13T06:11:20.525944+0000","last_became_active":"2020-07-13T06:11:20.525500+0000","last_became_peered":"2020-07-13T06:11:20.525500+0000","last_unstale":"2020-07-13T06:11:20.525944+0000","last_undegraded":"2020-07-13T06:11:20.525944+0000","last_fullsized":"2020-07-13T06:11:20.525944+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_clean_scrub_stamp":"2020-07-13T06:11:19.491245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:11:20.523505+0000","last_change":"2020-07-13T06:11:20.522830+0000","last_active":"2020-07-13T06:11:20.523505+0000","last_peered":"2020-07-13T06:11:20.523505+0000","last_clean":"2020-07-13T06:11:20.523505+0000","last_became_active":"2020-07-13T06:11:20.521765+0000","last_became_peered":"2020-07-13T06:11:20.521765+0000","last_unstale":"2020-07-13T06:11:20.523505+0000","last_undegraded":"2020-07-13T06:11:20.523505+0000","last_fullsized":"2020-07-13T06:11:20.523505+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_clean_scrub_stamp":"2020-07-13T06:11:19.491245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.516}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1140000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]}]}]}} 2020-07-13T06:11:25.244 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2020-07-13T06:11:25.512 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:11:25.514 INFO:teuthology.orchestra.run.smithi014.stderr:dumped all 2020-07-13T06:11:25.525 INFO:teuthology.orchestra.run.smithi014.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2020-07-13T06:11:25.450556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.932025"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:11:20.523287+0000","last_change":"2020-07-13T06:11:20.522277+0000","last_active":"2020-07-13T06:11:20.523287+0000","last_peered":"2020-07-13T06:11:20.523287+0000","last_clean":"2020-07-13T06:11:20.523287+0000","last_became_active":"2020-07-13T06:11:20.521514+0000","last_became_peered":"2020-07-13T06:11:20.521514+0000","last_unstale":"2020-07-13T06:11:20.523287+0000","last_undegraded":"2020-07-13T06:11:20.523287+0000","last_fullsized":"2020-07-13T06:11:20.523287+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_clean_scrub_stamp":"2020-07-13T06:11:19.491245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:11:20.522899+0000","last_change":"2020-07-13T06:11:20.521335+0000","last_active":"2020-07-13T06:11:20.522899+0000","last_peered":"2020-07-13T06:11:20.522899+0000","last_clean":"2020-07-13T06:11:20.522899+0000","last_became_active":"2020-07-13T06:11:20.520872+0000","last_became_peered":"2020-07-13T06:11:20.520872+0000","last_unstale":"2020-07-13T06:11:20.522899+0000","last_undegraded":"2020-07-13T06:11:20.522899+0000","last_fullsized":"2020-07-13T06:11:20.522899+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_clean_scrub_stamp":"2020-07-13T06:11:19.491245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:11:20.527873+0000","last_change":"2020-07-13T06:11:20.527270+0000","last_active":"2020-07-13T06:11:20.527873+0000","last_peered":"2020-07-13T06:11:20.527873+0000","last_clean":"2020-07-13T06:11:20.527873+0000","last_became_active":"2020-07-13T06:11:20.526889+0000","last_became_peered":"2020-07-13T06:11:20.526889+0000","last_unstale":"2020-07-13T06:11:20.527873+0000","last_undegraded":"2020-07-13T06:11:20.527873+0000","last_fullsized":"2020-07-13T06:11:20.527873+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_clean_scrub_stamp":"2020-07-13T06:11:19.491245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:11:20.524270+0000","last_change":"2020-07-13T06:11:20.524270+0000","last_active":"2020-07-13T06:11:20.524270+0000","last_peered":"2020-07-13T06:11:20.524270+0000","last_clean":"2020-07-13T06:11:20.524270+0000","last_became_active":"2020-07-13T06:11:20.523826+0000","last_became_peered":"2020-07-13T06:11:20.523826+0000","last_unstale":"2020-07-13T06:11:20.524270+0000","last_undegraded":"2020-07-13T06:11:20.524270+0000","last_fullsized":"2020-07-13T06:11:20.524270+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_clean_scrub_stamp":"2020-07-13T06:11:19.491245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:11:20.518676+0000","last_change":"2020-07-13T06:11:16.628962+0000","last_active":"2020-07-13T06:11:20.518676+0000","last_peered":"2020-07-13T06:11:20.518676+0000","last_clean":"2020-07-13T06:11:20.518676+0000","last_became_active":"2020-07-13T06:11:16.627429+0000","last_became_peered":"2020-07-13T06:11:16.627429+0000","last_unstale":"2020-07-13T06:11:20.518676+0000","last_undegraded":"2020-07-13T06:11:20.518676+0000","last_fullsized":"2020-07-13T06:11:20.518676+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:11:14.455048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:11:14.455048+0000","last_clean_scrub_stamp":"2020-07-13T06:11:14.455048+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:11:20.525639+0000","last_change":"2020-07-13T06:11:20.524616+0000","last_active":"2020-07-13T06:11:20.525639+0000","last_peered":"2020-07-13T06:11:20.525639+0000","last_clean":"2020-07-13T06:11:20.525639+0000","last_became_active":"2020-07-13T06:11:20.524129+0000","last_became_peered":"2020-07-13T06:11:20.524129+0000","last_unstale":"2020-07-13T06:11:20.525639+0000","last_undegraded":"2020-07-13T06:11:20.525639+0000","last_fullsized":"2020-07-13T06:11:20.525639+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_clean_scrub_stamp":"2020-07-13T06:11:19.491245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:11:20.523563+0000","last_change":"2020-07-13T06:11:20.523563+0000","last_active":"2020-07-13T06:11:20.523563+0000","last_peered":"2020-07-13T06:11:20.523563+0000","last_clean":"2020-07-13T06:11:20.523563+0000","last_became_active":"2020-07-13T06:11:20.522980+0000","last_became_peered":"2020-07-13T06:11:20.522980+0000","last_unstale":"2020-07-13T06:11:20.523563+0000","last_undegraded":"2020-07-13T06:11:20.523563+0000","last_fullsized":"2020-07-13T06:11:20.523563+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_clean_scrub_stamp":"2020-07-13T06:11:19.491245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:11:20.525944+0000","last_change":"2020-07-13T06:11:20.525944+0000","last_active":"2020-07-13T06:11:20.525944+0000","last_peered":"2020-07-13T06:11:20.525944+0000","last_clean":"2020-07-13T06:11:20.525944+0000","last_became_active":"2020-07-13T06:11:20.525500+0000","last_became_peered":"2020-07-13T06:11:20.525500+0000","last_unstale":"2020-07-13T06:11:20.525944+0000","last_undegraded":"2020-07-13T06:11:20.525944+0000","last_fullsized":"2020-07-13T06:11:20.525944+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_clean_scrub_stamp":"2020-07-13T06:11:19.491245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-13T06:11:20.523505+0000","last_change":"2020-07-13T06:11:20.522830+0000","last_active":"2020-07-13T06:11:20.523505+0000","last_peered":"2020-07-13T06:11:20.523505+0000","last_clean":"2020-07-13T06:11:20.523505+0000","last_became_active":"2020-07-13T06:11:20.521765+0000","last_became_peered":"2020-07-13T06:11:20.521765+0000","last_unstale":"2020-07-13T06:11:20.523505+0000","last_undegraded":"2020-07-13T06:11:20.523505+0000","last_fullsized":"2020-07-13T06:11:20.523505+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-13T06:11:19.491245+0000","last_clean_scrub_stamp":"2020-07-13T06:11:19.491245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.516}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1140000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]}]}]}} 2020-07-13T06:11:25.526 INFO:tasks.ceph.ceph_manager.c2:clean! 2020-07-13T06:11:25.527 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2020-07-13T06:11:25.527 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2020-07-13T06:11:25.527 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early health --format=json 2020-07-13T06:11:25.840 INFO:teuthology.orchestra.run.smithi014.stdout: 2020-07-13T06:11:25.859 INFO:teuthology.orchestra.run.smithi014.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-07-13T06:11:25.860 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2020-07-13T06:11:25.860 INFO:teuthology.run_tasks:Running task rgw... 2020-07-13T06:11:25.885 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2020-07-13T06:11:25.886 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2020-07-13T06:11:25.886 INFO:tasks.rgw:Creating data pools 2020-07-13T06:11:25.887 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2020-07-13T06:11:25.892 INFO:teuthology.orchestra.run.smithi017:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-07-13T06:11:26.483 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'default.rgw.buckets.data' created 2020-07-13T06:11:26.499 INFO:teuthology.orchestra.run.smithi017:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-07-13T06:11:27.491 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-07-13T06:11:27.518 INFO:teuthology.orchestra.run.smithi017:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-07-13T06:11:28.507 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'default.rgw.buckets.index' created 2020-07-13T06:11:28.519 INFO:teuthology.orchestra.run.smithi017:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-07-13T06:11:29.515 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-07-13T06:11:29.537 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2020-07-13T06:11:29.537 INFO:teuthology.orchestra.run.smithi017:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-07-13T06:11:29.797 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'default.rgw.buckets.data' already exists 2020-07-13T06:11:29.808 INFO:teuthology.orchestra.run.smithi017:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-07-13T06:11:30.512 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-07-13T06:11:30.526 INFO:teuthology.orchestra.run.smithi017:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-07-13T06:11:30.790 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'default.rgw.buckets.index' already exists 2020-07-13T06:11:30.800 INFO:teuthology.orchestra.run.smithi017:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-07-13T06:11:31.515 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-07-13T06:11:31.527 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2020-07-13T06:11:31.528 INFO:teuthology.orchestra.run.smithi014:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-07-13T06:11:32.543 INFO:teuthology.orchestra.run.smithi014.stderr:pool 'default.rgw.buckets.data' created 2020-07-13T06:11:32.556 INFO:teuthology.orchestra.run.smithi014:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-07-13T06:11:33.562 INFO:teuthology.orchestra.run.smithi014.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-07-13T06:11:33.590 INFO:teuthology.orchestra.run.smithi014:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-07-13T06:11:34.546 INFO:teuthology.orchestra.run.smithi014.stderr:pool 'default.rgw.buckets.index' created 2020-07-13T06:11:34.562 INFO:teuthology.orchestra.run.smithi014:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-07-13T06:11:35.566 INFO:teuthology.orchestra.run.smithi014.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-07-13T06:11:35.591 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2020-07-13T06:11:35.592 INFO:teuthology.orchestra.run.smithi014:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-07-13T06:11:35.861 INFO:teuthology.orchestra.run.smithi014.stderr:pool 'default.rgw.buckets.data' already exists 2020-07-13T06:11:35.872 INFO:teuthology.orchestra.run.smithi014:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-07-13T06:11:36.569 INFO:teuthology.orchestra.run.smithi014.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-07-13T06:11:36.581 INFO:teuthology.orchestra.run.smithi014:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-07-13T06:11:36.867 INFO:teuthology.orchestra.run.smithi014.stderr:pool 'default.rgw.buckets.index' already exists 2020-07-13T06:11:36.877 INFO:teuthology.orchestra.run.smithi014:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-07-13T06:11:37.572 INFO:teuthology.orchestra.run.smithi014.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-07-13T06:11:37.583 DEBUG:tasks.rgw:Pools created 2020-07-13T06:11:37.583 INFO:tasks.rgw:Configuring compression type = random 2020-07-13T06:11:37.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2020-07-13T06:11:37.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2020-07-13T06:11:37.585 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2020-07-13T06:11:37.624 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:11:37.624 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:11:37.636 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:37.635+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:39.475 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.471+0000 7efdfc6660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:39.475 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.471+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:39.476 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.475+0000 7efdfc6660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:39.476 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.475+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:39.477 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.475+0000 7efdfc6660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:39.478 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.475+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:39.478 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.475+0000 7efdfc6660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:39.501 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.499+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:39.502 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.499+0000 7efdfc6660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:39.503 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.499+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:39.503 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.499+0000 7efdfc6660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:39.503 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.499+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:39.503 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.499+0000 7efdfc6660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:39.503 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.499+0000 7efdfc6660c0 10 cannot find current period zonegroup using local zonegroup 2020-07-13T06:11:39.504 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.499+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:39.504 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.499+0000 7efdfc6660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:39.504 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.499+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:39.505 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.499+0000 7efdfc6660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:39.505 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.499+0000 7efdfc6660c0 10 Creating default zonegroup 2020-07-13T06:11:39.505 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.499+0000 7efdfc6660c0 20 get_system_obj_state: rctx=0x7ffd0de8aae0 obj=:.pools.avail state=0x56416d887f40 s->prefetch_data=0 2020-07-13T06:11:39.505 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.503+0000 7efdfc6660c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-07-13T06:11:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.515+0000 7efdfc6660c0 10 WARNING: store->list_zones() returned r=-2 2020-07-13T06:11:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.519+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.519+0000 7efdfc6660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:39.525 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.523+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:39.525 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.523+0000 7efdfc6660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:39.526 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.523+0000 7efdfc6660c0 10 could not read realm id: (2) No such file or directory 2020-07-13T06:11:39.526 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.523+0000 7efdfc6660c0 10 WARNING: failed to set zone as default, r=-22 2020-07-13T06:11:39.526 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.523+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.523+0000 7efdfc6660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.531+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:39.534 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.531+0000 7efdfc6660c0 20 rados_obj.operate() r=0 bl.length=839 2020-07-13T06:11:39.535 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.531+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:39.535 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.535+0000 7efdfc6660c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-13T06:11:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.535+0000 7efdfc6660c0 20 zonegroup default 2020-07-13T06:11:39.536 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.535+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:39.537 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.535+0000 7efdfc6660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:39.537 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.535+0000 7efdfc6660c0 10 Cannot find current period zone using local zone 2020-07-13T06:11:39.537 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.535+0000 7efdfc6660c0 10 Using default name default 2020-07-13T06:11:39.537 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.535+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:39.543 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.535+0000 7efdfc6660c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:39.543 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.535+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:39.543 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.535+0000 7efdfc6660c0 20 rados_obj.operate() r=0 bl.length=839 2020-07-13T06:11:39.543 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.535+0000 7efdfc6660c0 20 zone default found 2020-07-13T06:11:39.544 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:39.535+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:41.488 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:41.487+0000 7efdfc6660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:41.488 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:41.487+0000 7efdfc6660c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:41.489 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:41.487+0000 7efdfc6660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:41.490 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:41.487+0000 7efdfc6660c0 20 started sync module instance, tier type = 2020-07-13T06:11:41.490 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:41.487+0000 7efdfc6660c0 20 started zone id=2bfa87c0-41d8-4d7f-a74f-13751960dc09 (name=default) with tier type = 2020-07-13T06:11:43.545 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:43.543+0000 7efdfc6660c0 20 add_watcher() i=0 2020-07-13T06:11:43.546 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:43.543+0000 7efdfc6660c0 20 add_watcher() i=1 2020-07-13T06:11:43.546 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:43.543+0000 7efdfc6660c0 20 add_watcher() i=2 2020-07-13T06:11:43.546 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:43.543+0000 7efdfc6660c0 20 add_watcher() i=3 2020-07-13T06:11:43.547 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:43.543+0000 7efdfc6660c0 20 add_watcher() i=4 2020-07-13T06:11:43.547 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:43.543+0000 7efdfc6660c0 20 add_watcher() i=5 2020-07-13T06:11:43.547 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:43.543+0000 7efdfc6660c0 20 add_watcher() i=6 2020-07-13T06:11:43.548 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:43.547+0000 7efdfc6660c0 20 add_watcher() i=7 2020-07-13T06:11:43.548 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:43.547+0000 7efdfc6660c0 2 all 8 watchers are set, enabling cache 2020-07-13T06:11:43.548 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:43.547+0000 7efdfc6660c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-13T06:11:43.548 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:43.547+0000 7efdfc6660c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-13T06:11:43.549 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:43.547+0000 7efdb57ea700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-13T06:11:47.519 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.519+0000 7efdfc6660c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-07-13T06:11:47.631 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.627+0000 7efd1bfff700 20 reqs_thread_entry: start 2020-07-13T06:11:47.631 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.627+0000 7efd19ffb700 20 reqs_thread_entry: start 2020-07-13T06:11:47.631 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.627+0000 7efdfc6660c0 20 init_complete bucket index max shards: 11 2020-07-13T06:11:47.632 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.631+0000 7efd07fff700 20 reqs_thread_entry: start 2020-07-13T06:11:47.659 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.655+0000 7efdb57ea700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-13T06:11:47.662 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.659+0000 7efdfc6660c0 20 remove_watcher() i=0 2020-07-13T06:11:47.662 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.659+0000 7efdfc6660c0 2 removed watcher, disabling cache 2020-07-13T06:11:47.665 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.663+0000 7efdfc6660c0 20 remove_watcher() i=1 2020-07-13T06:11:47.668 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.667+0000 7efdfc6660c0 20 remove_watcher() i=2 2020-07-13T06:11:47.671 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.667+0000 7efdfc6660c0 20 remove_watcher() i=3 2020-07-13T06:11:47.674 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.671+0000 7efdfc6660c0 20 remove_watcher() i=4 2020-07-13T06:11:47.676 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.675+0000 7efdfc6660c0 20 remove_watcher() i=5 2020-07-13T06:11:47.679 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.679+0000 7efdfc6660c0 20 remove_watcher() i=6 2020-07-13T06:11:47.682 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.679+0000 7efdfc6660c0 20 remove_watcher() i=7 2020-07-13T06:11:47.690 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2020-07-13T06:11:47.690 DEBUG:tasks.util.rgw: json result: [] 2020-07-13T06:11:47.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-13T06:11:47.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-13T06:11:47.692 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-07-13T06:11:47.778 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:11:47.779 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:11:47.791 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.787+0000 7fbbd6d3f0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-13T06:11:47.792 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.787+0000 7fbbd6d3f0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-13T06:11:47.799 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.795+0000 7fbafffff700 20 reqs_thread_entry: start 2020-07-13T06:11:47.799 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.795+0000 7fbbd6d3f0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.800 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.799+0000 7fbbd6d3f0c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:47.801 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.799+0000 7fbbd6d3f0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.802 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.799+0000 7fbbd6d3f0c0 20 rados_obj.operate() r=0 bl.length=839 2020-07-13T06:11:47.802 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.799+0000 7fbbd6d3f0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.802 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.799+0000 7fbbd6d3f0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:47.803 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.799+0000 7fbbd6d3f0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.803 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.803+0000 7fbbd6d3f0c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:47.803 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.803+0000 7fbbd6d3f0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.804 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.803+0000 7fbbd6d3f0c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-13T06:11:47.814 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"2bfa87c0-41d8-4d7f-a74f-13751960dc09","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-07-13T06:11:47.814 DEBUG:tasks.util.rgw: json result: {'id': '2bfa87c0-41d8-4d7f-a74f-13751960dc09', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-07-13T06:11:47.815 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2020-07-13T06:11:47.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2020-07-13T06:11:47.815 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2020-07-13T06:11:47.851 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:11:47.851 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:11:47.862 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.859+0000 7f39a7e240c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.864 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.863+0000 7f39a7e240c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:47.865 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.863+0000 7f39a7e240c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.865 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.863+0000 7f39a7e240c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:47.865 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.863+0000 7f39a7e240c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.866 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.863+0000 7f39a7e240c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:47.866 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.863+0000 7f39a7e240c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.867 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.863+0000 7f39a7e240c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:47.893 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.891+0000 7f39a7e240c0 20 RGWRados::pool_iterate: got zonegroup_info.0e0c4ddf-32b1-40ec-9de6-083f0a0dbf3e 2020-07-13T06:11:47.893 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.891+0000 7f39a7e240c0 20 RGWRados::pool_iterate: got zone_info.2bfa87c0-41d8-4d7f-a74f-13751960dc09 2020-07-13T06:11:47.893 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.891+0000 7f39a7e240c0 20 RGWRados::pool_iterate: got zone_names.default 2020-07-13T06:11:47.894 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.891+0000 7f39a7e240c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-07-13T06:11:47.894 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.891+0000 7f39a7e240c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.894 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.891+0000 7f39a7e240c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:47.895 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.891+0000 7f39a7e240c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.895 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.891+0000 7f39a7e240c0 20 rados_obj.operate() r=0 bl.length=855 2020-07-13T06:11:47.895 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.891+0000 7f39a7e240c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.896 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.895+0000 7f39a7e240c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:47.896 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.895+0000 7f39a7e240c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.896 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.895+0000 7f39a7e240c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-13T06:11:47.897 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.895+0000 7f39a7e240c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.897 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.895+0000 7f39a7e240c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:47.897 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.895+0000 7f39a7e240c0 10 cannot find current period zonegroup using local zonegroup 2020-07-13T06:11:47.898 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.895+0000 7f39a7e240c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.898 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.895+0000 7f39a7e240c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:47.898 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.895+0000 7f39a7e240c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.899 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.895+0000 7f39a7e240c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:47.899 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.895+0000 7f39a7e240c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.899 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.899+0000 7f39a7e240c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-13T06:11:47.900 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.899+0000 7f39a7e240c0 20 zonegroup default 2020-07-13T06:11:47.900 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.899+0000 7f39a7e240c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.901 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.899+0000 7f39a7e240c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:47.901 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.899+0000 7f39a7e240c0 10 Cannot find current period zone using local zone 2020-07-13T06:11:47.901 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.899+0000 7f39a7e240c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.901 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.899+0000 7f39a7e240c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:47.901 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.899+0000 7f39a7e240c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.902 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.899+0000 7f39a7e240c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:47.902 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.899+0000 7f39a7e240c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.902 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.899+0000 7f39a7e240c0 20 rados_obj.operate() r=0 bl.length=855 2020-07-13T06:11:47.903 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.899+0000 7f39a7e240c0 20 zone default found 2020-07-13T06:11:47.903 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.899+0000 7f39a7e240c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.903 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.899+0000 7f39a7e240c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:47.904 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.899+0000 7f39a7e240c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:47.904 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.903+0000 7f39a7e240c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:47.904 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.903+0000 7f39a7e240c0 20 started sync module instance, tier type = 2020-07-13T06:11:47.904 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.903+0000 7f39a7e240c0 20 started zone id=2bfa87c0-41d8-4d7f-a74f-13751960dc09 (name=default) with tier type = 2020-07-13T06:11:47.932 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.923+0000 7f39a7e240c0 20 add_watcher() i=0 2020-07-13T06:11:47.932 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.923+0000 7f39a7e240c0 20 add_watcher() i=1 2020-07-13T06:11:47.933 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.923+0000 7f39a7e240c0 20 add_watcher() i=2 2020-07-13T06:11:47.933 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.923+0000 7f39a7e240c0 20 add_watcher() i=3 2020-07-13T06:11:47.933 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.923+0000 7f39a7e240c0 20 add_watcher() i=4 2020-07-13T06:11:47.933 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.923+0000 7f39a7e240c0 20 add_watcher() i=5 2020-07-13T06:11:47.933 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.923+0000 7f39a7e240c0 20 add_watcher() i=6 2020-07-13T06:11:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.927+0000 7f39a7e240c0 20 add_watcher() i=7 2020-07-13T06:11:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.927+0000 7f39a7e240c0 2 all 8 watchers are set, enabling cache 2020-07-13T06:11:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.927+0000 7f39a7e240c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-13T06:11:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.927+0000 7f39a7e240c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-13T06:11:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:47.927+0000 7f3960fe9700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-13T06:11:48.023 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.019+0000 7f38cb7fe700 20 reqs_thread_entry: start 2020-07-13T06:11:48.024 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.023+0000 7f39a7e240c0 20 init_complete bucket index max shards: 11 2020-07-13T06:11:48.024 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.023+0000 7f38c97fa700 20 reqs_thread_entry: start 2020-07-13T06:11:48.024 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.023+0000 7f38bb7fe700 20 reqs_thread_entry: start 2020-07-13T06:11:48.052 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.051+0000 7f3960fe9700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-13T06:11:48.054 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.051+0000 7f39a7e240c0 20 remove_watcher() i=0 2020-07-13T06:11:48.055 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.051+0000 7f39a7e240c0 2 removed watcher, disabling cache 2020-07-13T06:11:48.057 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.055+0000 7f39a7e240c0 20 remove_watcher() i=1 2020-07-13T06:11:48.060 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.059+0000 7f39a7e240c0 20 remove_watcher() i=2 2020-07-13T06:11:48.062 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.059+0000 7f39a7e240c0 20 remove_watcher() i=3 2020-07-13T06:11:48.065 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.063+0000 7f39a7e240c0 20 remove_watcher() i=4 2020-07-13T06:11:48.068 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.067+0000 7f39a7e240c0 20 remove_watcher() i=5 2020-07-13T06:11:48.071 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.067+0000 7f39a7e240c0 20 remove_watcher() i=6 2020-07-13T06:11:48.073 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.071+0000 7f39a7e240c0 20 remove_watcher() i=7 2020-07-13T06:11:48.080 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2020-07-13T06:11:48.081 DEBUG:tasks.util.rgw: json result: [] 2020-07-13T06:11:48.081 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-13T06:11:48.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-13T06:11:48.082 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-07-13T06:11:48.166 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:11:48.166 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:11:48.177 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.175+0000 7f97a65da0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-13T06:11:48.177 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.175+0000 7f97a65da0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-13T06:11:48.185 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.183+0000 7f97a65da0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:48.185 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.183+0000 7f96cf7fe700 20 reqs_thread_entry: start 2020-07-13T06:11:48.186 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.183+0000 7f97a65da0c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:48.187 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.183+0000 7f97a65da0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:48.188 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.187+0000 7f97a65da0c0 20 rados_obj.operate() r=0 bl.length=855 2020-07-13T06:11:48.188 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.187+0000 7f97a65da0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:48.189 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.187+0000 7f97a65da0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:48.189 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.187+0000 7f97a65da0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:48.190 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.187+0000 7f97a65da0c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:48.190 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.187+0000 7f97a65da0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:48.191 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:11:48.187+0000 7f97a65da0c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-13T06:11:48.200 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"2bfa87c0-41d8-4d7f-a74f-13751960dc09","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-07-13T06:11:48.201 DEBUG:tasks.util.rgw: json result: {'id': '2bfa87c0-41d8-4d7f-a74f-13751960dc09', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-07-13T06:11:48.201 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2020-07-13T06:11:48.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2020-07-13T06:11:48.202 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2020-07-13T06:11:48.238 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:11:48.238 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:11:48.248 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:48.249+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:49.546 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.549+0000 7f8a1b33d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:49.546 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.549+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:49.555 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.557+0000 7f8a1b33d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:49.556 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.557+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:49.556 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.557+0000 7f8a1b33d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:49.557 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.557+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:49.559 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.557+0000 7f8a1b33d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:49.577 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.577+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:49.578 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.581+0000 7f8a1b33d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:49.578 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.581+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:49.578 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.581+0000 7f8a1b33d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:49.578 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.581+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:49.579 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.581+0000 7f8a1b33d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:49.580 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.581+0000 7f8a1b33d0c0 10 cannot find current period zonegroup using local zonegroup 2020-07-13T06:11:49.580 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.581+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:49.580 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.581+0000 7f8a1b33d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:49.580 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.581+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:49.581 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.581+0000 7f8a1b33d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:49.581 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.581+0000 7f8a1b33d0c0 10 Creating default zonegroup 2020-07-13T06:11:49.581 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.581+0000 7f8a1b33d0c0 20 get_system_obj_state: rctx=0x7fff52e6b440 obj=:.pools.avail state=0x561d36f60f40 s->prefetch_data=0 2020-07-13T06:11:49.581 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.581+0000 7f8a1b33d0c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-07-13T06:11:49.597 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.601+0000 7f8a1b33d0c0 10 WARNING: store->list_zones() returned r=-2 2020-07-13T06:11:49.598 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.601+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:49.598 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.601+0000 7f8a1b33d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:49.603 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.605+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:49.604 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.605+0000 7f8a1b33d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:49.604 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.605+0000 7f8a1b33d0c0 10 could not read realm id: (2) No such file or directory 2020-07-13T06:11:49.605 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.605+0000 7f8a1b33d0c0 10 WARNING: failed to set zone as default, r=-22 2020-07-13T06:11:49.605 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.605+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:49.606 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.605+0000 7f8a1b33d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:49.610 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.613+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:49.611 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.613+0000 7f8a1b33d0c0 20 rados_obj.operate() r=0 bl.length=839 2020-07-13T06:11:49.611 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.613+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:49.613 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.613+0000 7f8a1b33d0c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-13T06:11:49.613 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.613+0000 7f8a1b33d0c0 20 zonegroup default 2020-07-13T06:11:49.614 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.613+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:49.614 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.613+0000 7f8a1b33d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:49.614 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.613+0000 7f8a1b33d0c0 10 Cannot find current period zone using local zone 2020-07-13T06:11:49.615 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.613+0000 7f8a1b33d0c0 10 Using default name default 2020-07-13T06:11:49.615 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.613+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:49.616 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.617+0000 7f8a1b33d0c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:49.616 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.617+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:49.616 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.617+0000 7f8a1b33d0c0 20 rados_obj.operate() r=0 bl.length=839 2020-07-13T06:11:49.616 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.617+0000 7f8a1b33d0c0 20 zone default found 2020-07-13T06:11:49.617 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:49.617+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:51.567 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:51.565+0000 7f8a1b33d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:51.567 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:51.565+0000 7f8a1b33d0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:51.583 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:51.585+0000 7f8a1b33d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:51.584 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:51.585+0000 7f8a1b33d0c0 20 started sync module instance, tier type = 2020-07-13T06:11:51.584 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:51.585+0000 7f8a1b33d0c0 20 started zone id=604cd5af-a22c-49ea-b5b8-912d3c8ecf82 (name=default) with tier type = 2020-07-13T06:11:53.613 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:53.613+0000 7f8a1b33d0c0 20 add_watcher() i=0 2020-07-13T06:11:53.614 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:53.613+0000 7f8a1b33d0c0 20 add_watcher() i=1 2020-07-13T06:11:53.614 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:53.613+0000 7f8a1b33d0c0 20 add_watcher() i=2 2020-07-13T06:11:53.616 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:53.613+0000 7f8a1b33d0c0 20 add_watcher() i=3 2020-07-13T06:11:53.616 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:53.613+0000 7f8a1b33d0c0 20 add_watcher() i=4 2020-07-13T06:11:53.616 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:53.613+0000 7f8a1b33d0c0 20 add_watcher() i=5 2020-07-13T06:11:53.617 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:53.613+0000 7f8a1b33d0c0 20 add_watcher() i=6 2020-07-13T06:11:53.618 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:53.617+0000 7f8a1b33d0c0 20 add_watcher() i=7 2020-07-13T06:11:53.618 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:53.617+0000 7f8a1b33d0c0 2 all 8 watchers are set, enabling cache 2020-07-13T06:11:53.618 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:53.617+0000 7f8a1b33d0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-13T06:11:53.619 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:53.617+0000 7f8a1b33d0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-13T06:11:53.619 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:53.617+0000 7f894ffff700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-13T06:11:57.583 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.584+0000 7f8a1b33d0c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-07-13T06:11:57.690 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.688+0000 7f893effd700 20 reqs_thread_entry: start 2020-07-13T06:11:57.690 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.688+0000 7f893cff9700 20 reqs_thread_entry: start 2020-07-13T06:11:57.691 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.688+0000 7f8a1b33d0c0 20 init_complete bucket index max shards: 11 2020-07-13T06:11:57.691 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.688+0000 7f892affd700 20 reqs_thread_entry: start 2020-07-13T06:11:57.718 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.716+0000 7f894ffff700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-13T06:11:57.720 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.720+0000 7f8a1b33d0c0 20 remove_watcher() i=0 2020-07-13T06:11:57.721 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.720+0000 7f8a1b33d0c0 2 removed watcher, disabling cache 2020-07-13T06:11:57.723 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.724+0000 7f8a1b33d0c0 20 remove_watcher() i=1 2020-07-13T06:11:57.726 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.724+0000 7f8a1b33d0c0 20 remove_watcher() i=2 2020-07-13T06:11:57.729 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.728+0000 7f8a1b33d0c0 20 remove_watcher() i=3 2020-07-13T06:11:57.732 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.732+0000 7f8a1b33d0c0 20 remove_watcher() i=4 2020-07-13T06:11:57.735 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.736+0000 7f8a1b33d0c0 20 remove_watcher() i=5 2020-07-13T06:11:57.738 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.736+0000 7f8a1b33d0c0 20 remove_watcher() i=6 2020-07-13T06:11:57.741 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.740+0000 7f8a1b33d0c0 20 remove_watcher() i=7 2020-07-13T06:11:57.749 INFO:teuthology.orchestra.run.smithi014.stdout:[] 2020-07-13T06:11:57.749 DEBUG:tasks.util.rgw: json result: [] 2020-07-13T06:11:57.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-13T06:11:57.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-13T06:11:57.751 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-07-13T06:11:57.832 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:11:57.833 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:11:57.846 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.844+0000 7f37fadb10c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-13T06:11:57.847 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.844+0000 7f37fadb10c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-13T06:11:57.854 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.852+0000 7f37fadb10c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:57.854 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.852+0000 7f372bfff700 20 reqs_thread_entry: start 2020-07-13T06:11:57.855 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.856+0000 7f37fadb10c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:57.855 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.856+0000 7f37fadb10c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:57.857 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.856+0000 7f37fadb10c0 20 rados_obj.operate() r=0 bl.length=839 2020-07-13T06:11:57.857 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.856+0000 7f37fadb10c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:57.858 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.856+0000 7f37fadb10c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:57.858 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.856+0000 7f37fadb10c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:57.859 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.860+0000 7f37fadb10c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:57.859 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.860+0000 7f37fadb10c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:57.860 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.860+0000 7f37fadb10c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-13T06:11:57.872 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"604cd5af-a22c-49ea-b5b8-912d3c8ecf82","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-07-13T06:11:57.872 DEBUG:tasks.util.rgw: json result: {'id': '604cd5af-a22c-49ea-b5b8-912d3c8ecf82', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-07-13T06:11:57.873 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2020-07-13T06:11:57.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2020-07-13T06:11:57.874 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2020-07-13T06:11:57.951 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:11:57.952 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:11:57.960 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.960+0000 7f37269bf0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:57.963 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.964+0000 7f37269bf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:57.964 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.964+0000 7f37269bf0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:57.965 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.964+0000 7f37269bf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:57.965 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.964+0000 7f37269bf0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:57.965 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.964+0000 7f37269bf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:57.966 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.964+0000 7f37269bf0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:57.966 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.964+0000 7f37269bf0c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:57.989 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.988+0000 7f37269bf0c0 20 RGWRados::pool_iterate: got zonegroup_info.baeaf3a2-1111-4d9a-9efe-3be2820cfda3 2020-07-13T06:11:57.989 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.988+0000 7f37269bf0c0 20 RGWRados::pool_iterate: got zone_names.default 2020-07-13T06:11:57.990 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.988+0000 7f37269bf0c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-07-13T06:11:57.990 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.988+0000 7f37269bf0c0 20 RGWRados::pool_iterate: got zone_info.604cd5af-a22c-49ea-b5b8-912d3c8ecf82 2020-07-13T06:11:57.990 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.988+0000 7f37269bf0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:57.991 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.988+0000 7f37269bf0c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:57.991 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.988+0000 7f37269bf0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:57.991 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.988+0000 7f37269bf0c0 20 rados_obj.operate() r=0 bl.length=855 2020-07-13T06:11:57.992 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.988+0000 7f37269bf0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:57.992 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.992+0000 7f37269bf0c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:57.992 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.992+0000 7f37269bf0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:57.993 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.992+0000 7f37269bf0c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-13T06:11:57.993 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.992+0000 7f37269bf0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:57.994 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.992+0000 7f37269bf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:57.994 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.992+0000 7f37269bf0c0 10 cannot find current period zonegroup using local zonegroup 2020-07-13T06:11:57.994 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.992+0000 7f37269bf0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:57.994 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.992+0000 7f37269bf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:57.994 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.992+0000 7f37269bf0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:57.995 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.992+0000 7f37269bf0c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:57.996 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.992+0000 7f37269bf0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:57.996 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.992+0000 7f37269bf0c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-13T06:11:57.996 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.992+0000 7f37269bf0c0 20 zonegroup default 2020-07-13T06:11:57.996 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.992+0000 7f37269bf0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:57.999 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.996+0000 7f37269bf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:57.999 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.996+0000 7f37269bf0c0 10 Cannot find current period zone using local zone 2020-07-13T06:11:57.999 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.996+0000 7f37269bf0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:58.000 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.996+0000 7f37269bf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:58.001 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.996+0000 7f37269bf0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:58.001 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.996+0000 7f37269bf0c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:58.001 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.996+0000 7f37269bf0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:58.001 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.996+0000 7f37269bf0c0 20 rados_obj.operate() r=0 bl.length=855 2020-07-13T06:11:58.003 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.996+0000 7f37269bf0c0 20 zone default found 2020-07-13T06:11:58.004 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.996+0000 7f37269bf0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:58.004 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.996+0000 7f37269bf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:58.004 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:57.996+0000 7f37269bf0c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:58.004 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.000+0000 7f37269bf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:58.004 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.000+0000 7f37269bf0c0 20 started sync module instance, tier type = 2020-07-13T06:11:58.005 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.000+0000 7f37269bf0c0 20 started zone id=604cd5af-a22c-49ea-b5b8-912d3c8ecf82 (name=default) with tier type = 2020-07-13T06:11:58.028 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.028+0000 7f37269bf0c0 20 add_watcher() i=0 2020-07-13T06:11:58.028 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.028+0000 7f37269bf0c0 20 add_watcher() i=1 2020-07-13T06:11:58.028 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.028+0000 7f37269bf0c0 20 add_watcher() i=2 2020-07-13T06:11:58.029 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.028+0000 7f37269bf0c0 20 add_watcher() i=3 2020-07-13T06:11:58.029 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.028+0000 7f37269bf0c0 20 add_watcher() i=4 2020-07-13T06:11:58.029 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.028+0000 7f37269bf0c0 20 add_watcher() i=5 2020-07-13T06:11:58.030 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.028+0000 7f37269bf0c0 20 add_watcher() i=6 2020-07-13T06:11:58.031 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.028+0000 7f37269bf0c0 20 add_watcher() i=7 2020-07-13T06:11:58.031 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.028+0000 7f37269bf0c0 2 all 8 watchers are set, enabling cache 2020-07-13T06:11:58.031 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.028+0000 7f37269bf0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-13T06:11:58.032 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.028+0000 7f37269bf0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-13T06:11:58.032 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.032+0000 7f36dffe7700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-13T06:11:58.128 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.124+0000 7f364effd700 20 reqs_thread_entry: start 2020-07-13T06:11:58.128 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.124+0000 7f37269bf0c0 20 init_complete bucket index max shards: 11 2020-07-13T06:11:58.129 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.124+0000 7f364cff9700 20 reqs_thread_entry: start 2020-07-13T06:11:58.129 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.128+0000 7f363effd700 20 reqs_thread_entry: start 2020-07-13T06:11:58.161 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.152+0000 7f36dffe7700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-13T06:11:58.162 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.156+0000 7f37269bf0c0 20 remove_watcher() i=0 2020-07-13T06:11:58.162 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.156+0000 7f37269bf0c0 2 removed watcher, disabling cache 2020-07-13T06:11:58.162 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.160+0000 7f37269bf0c0 20 remove_watcher() i=1 2020-07-13T06:11:58.163 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.164+0000 7f37269bf0c0 20 remove_watcher() i=2 2020-07-13T06:11:58.166 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.164+0000 7f37269bf0c0 20 remove_watcher() i=3 2020-07-13T06:11:58.169 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.168+0000 7f37269bf0c0 20 remove_watcher() i=4 2020-07-13T06:11:58.172 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.172+0000 7f37269bf0c0 20 remove_watcher() i=5 2020-07-13T06:11:58.175 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.176+0000 7f37269bf0c0 20 remove_watcher() i=6 2020-07-13T06:11:58.178 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.176+0000 7f37269bf0c0 20 remove_watcher() i=7 2020-07-13T06:11:58.202 INFO:teuthology.orchestra.run.smithi014.stdout:[] 2020-07-13T06:11:58.203 DEBUG:tasks.util.rgw: json result: [] 2020-07-13T06:11:58.203 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-13T06:11:58.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-13T06:11:58.204 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-07-13T06:11:58.288 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:11:58.288 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:11:58.301 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.300+0000 7f5fcd4580c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-13T06:11:58.301 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.300+0000 7f5fcd4580c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-13T06:11:58.309 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.308+0000 7f5efdffb700 20 reqs_thread_entry: start 2020-07-13T06:11:58.309 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.308+0000 7f5fcd4580c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:58.310 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.308+0000 7f5fcd4580c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:58.311 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.308+0000 7f5fcd4580c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:58.313 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.312+0000 7f5fcd4580c0 20 rados_obj.operate() r=0 bl.length=855 2020-07-13T06:11:58.314 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.312+0000 7f5fcd4580c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:58.314 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.312+0000 7f5fcd4580c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-13T06:11:58.314 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.312+0000 7f5fcd4580c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:58.314 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.312+0000 7f5fcd4580c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-13T06:11:58.314 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.312+0000 7f5fcd4580c0 20 rados->read ofs=0 len=0 2020-07-13T06:11:58.315 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:11:58.312+0000 7f5fcd4580c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-13T06:11:58.322 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"604cd5af-a22c-49ea-b5b8-912d3c8ecf82","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-07-13T06:11:58.323 DEBUG:tasks.util.rgw: json result: {'id': '604cd5af-a22c-49ea-b5b8-912d3c8ecf82', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-07-13T06:11:58.323 INFO:tasks.rgw:Starting rgw... 2020-07-13T06:11:58.324 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2020-07-13T06:11:58.324 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-07-13T06:11:58.324 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-07-13T06:11:58.325 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-07-13T06:11:58.329 INFO:tasks.rgw.c1.client.0:Started 2020-07-13T06:11:58.329 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2020-07-13T06:11:58.330 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-07-13T06:11:58.330 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-07-13T06:11:58.331 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-07-13T06:11:58.334 INFO:tasks.rgw.c1.client.1:Started 2020-07-13T06:11:58.334 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2020-07-13T06:11:58.334 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-07-13T06:11:58.335 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-07-13T06:11:58.335 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-07-13T06:11:58.369 INFO:tasks.rgw.c2.client.0:Started 2020-07-13T06:11:58.370 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2020-07-13T06:11:58.370 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-07-13T06:11:58.370 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-07-13T06:11:58.371 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-07-13T06:11:58.373 INFO:tasks.rgw.c2.client.1:Started 2020-07-13T06:11:58.374 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi017.front.sepia.ceph.com:8000/ 2020-07-13T06:11:58.374 INFO:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8000/ 2020-07-13T06:11:58.393 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-13T06:11:58.393 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2020-07-13T06:11:58.396 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi017.front.sepia.ceph.com port 8000: Connection refused 2020-07-13T06:11:58.397 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-07-13T06:11:59.399 INFO:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8000/ 2020-07-13T06:11:59.414 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-13T06:11:59.415 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2020-07-13T06:11:59.419 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-07-13T06:11:59.420 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2020-07-13T06:11:59.421 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi017.front.sepia.ceph.com:8001/ 2020-07-13T06:11:59.421 INFO:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001/ 2020-07-13T06:11:59.478 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-13T06:11:59.479 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2020-07-13T06:11:59.483 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 53500 2020-07-13T06:11:59.484 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2020-07-13T06:11:59.484 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi014.front.sepia.ceph.com:8000/ 2020-07-13T06:11:59.485 INFO:teuthology.orchestra.run.smithi014:> curl http://smithi014.front.sepia.ceph.com:8000/ 2020-07-13T06:11:59.498 INFO:teuthology.orchestra.run.smithi014.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-13T06:11:59.498 INFO:teuthology.orchestra.run.smithi014.stderr: Dload Upload Total Spent Left Speed 2020-07-13T06:11:59.503 INFO:teuthology.orchestra.run.smithi014.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2020-07-13T06:11:59.505 INFO:teuthology.orchestra.run.smithi014.stdout:anonymous 2020-07-13T06:11:59.505 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi014.front.sepia.ceph.com:8001/ 2020-07-13T06:11:59.506 INFO:teuthology.orchestra.run.smithi014:> curl http://smithi014.front.sepia.ceph.com:8001/ 2020-07-13T06:11:59.561 INFO:teuthology.orchestra.run.smithi014.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-13T06:11:59.561 INFO:teuthology.orchestra.run.smithi014.stderr: Dload Upload Total Spent Left Speed 2020-07-13T06:11:59.567 INFO:teuthology.orchestra.run.smithi014.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-07-13T06:11:59.567 INFO:teuthology.orchestra.run.smithi014.stdout:anonymous 2020-07-13T06:11:59.568 INFO:teuthology.run_tasks:Running task rgw-multisite... 2020-07-13T06:12:00.075 INFO:tasks.rgw_multisite:creating realm.. 2020-07-13T06:12:00.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:00.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:00.077 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:12:00.112 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:12:00.116 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:12:00.164 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"3bffea65-3664-4393-a4a6-65c46ad5d668","name":"test-realm","current_period":"b2a1b5d3-8c44-40d3-9c92-45a62692c41d","epoch":1} 2020-07-13T06:12:00.164 INFO:tasks.rgw_multisite:creating master zonegroup.. 2020-07-13T06:12:00.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:00.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:00.166 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:12:00.245 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:12:00.246 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:12:00.271 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:12:00.262+0000 7f7486a490c0 -1 old format 2020-07-13T06:12:00.284 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","sync_policy":{"groups":[]}} 2020-07-13T06:12:00.285 INFO:tasks.rgw_multisite:creating master zone.. 2020-07-13T06:12:00.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'HVCB2URARHK5PDP7', '--secret', 'iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh', '--endpoints', 'http://smithi017.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:00.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'HVCB2URARHK5PDP7', '--secret', 'iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh', '--endpoints', 'http://smithi017.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:00.287 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key HVCB2URARHK5PDP7 --secret iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh --endpoints http://smithi017.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:12:00.374 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:12:00.374 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:12:00.457 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"5fd641b5-9375-4754-8474-97ebd79eafb6","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"HVCB2URARHK5PDP7","secret_key":"iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668"} 2020-07-13T06:12:00.457 INFO:teuthology.orchestra.run.smithi017:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2020-07-13T06:12:01.482 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'test-zone1.rgw.buckets.data' created 2020-07-13T06:12:01.498 INFO:teuthology.orchestra.run.smithi017:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2020-07-13T06:12:02.518 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2020-07-13T06:12:02.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:02.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:02.535 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:12:02.571 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:12:02.571 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:12:02.599 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"5fd641b5-9375-4754-8474-97ebd79eafb6","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"HVCB2URARHK5PDP7","secret_key":"iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-07-13T06:12:02.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:02.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:02.601 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:12:02.686 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:12:02.687 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:12:10.772 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"587bd98b-69d2-49ba-9dbd-2be72177aade","epoch":1,"predecessor_uuid":"b2a1b5d3-8c44-40d3-9c92-45a62692c41d","sync_status":[],"period_map":{"id":"587bd98b-69d2-49ba-9dbd-2be72177aade","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","zones":[{"id":"5fd641b5-9375-4754-8474-97ebd79eafb6","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5fd641b5-9375-4754-8474-97ebd79eafb6","val":1497020214}]},"master_zonegroup":"test-zonegroup","master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","realm_name":"test-realm","realm_epoch":2} 2020-07-13T06:12:10.774 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2020-07-13T06:12:10.775 INFO:teuthology.orchestra.run:waiting for 300 2020-07-13T06:12:10.821 INFO:tasks.rgw.c1.client.0.smithi017.stdout:2020-07-13T06:12:10.818+0000 7f5906a6a700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 13515) UID: 0 2020-07-13T06:12:10.821 INFO:tasks.rgw.c1.client.0.smithi017.stdout:2020-07-13T06:12:10.818+0000 7f591e141100 -1 shutting down 2020-07-13T06:12:16.345 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2020-07-13T06:12:16.880 INFO:tasks.rgw.c1.client.0:Stopped 2020-07-13T06:12:16.881 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-07-13T06:12:16.882 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-07-13T06:12:16.888 INFO:tasks.rgw.c1.client.0:Started 2020-07-13T06:12:16.889 INFO:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8000 2020-07-13T06:12:16.906 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-13T06:12:16.907 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2020-07-13T06:12:16.910 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi017.front.sepia.ceph.com port 8000: Connection refused 2020-07-13T06:12:16.911 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-07-13T06:12:17.913 INFO:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8000 2020-07-13T06:12:17.929 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-13T06:12:17.929 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2020-07-13T06:12:17.934 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-07-13T06:12:17.935 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2020-07-13T06:12:17.936 INFO:tasks.rgw_multisite:creating admin user.. 2020-07-13T06:12:17.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'HVCB2URARHK5PDP7', '--secret', 'iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:17.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'HVCB2URARHK5PDP7', '--secret', 'iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:17.937 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key HVCB2URARHK5PDP7 --secret iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:12:18.027 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:12:18.027 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:12:18.238 INFO:teuthology.orchestra.run.smithi017.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"HVCB2URARHK5PDP7","secret_key":"iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":"true","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2020-07-13T06:12:18.266 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-07-13T06:12:18.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi017.front.sepia.ceph.com:8000', '--access-key', 'HVCB2URARHK5PDP7', '--secret', 'iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:18.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi017.front.sepia.ceph.com:8000', '--access-key', 'HVCB2URARHK5PDP7', '--secret', 'iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:18.268 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi017.front.sepia.ceph.com:8000 --access-key HVCB2URARHK5PDP7 --secret iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:12:18.311 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:12:18.311 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:12:18.432 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"3bffea65-3664-4393-a4a6-65c46ad5d668","name":"test-realm","current_period":"587bd98b-69d2-49ba-9dbd-2be72177aade","epoch":2} 2020-07-13T06:12:18.433 INFO:tasks.rgw_multisite:creating zone.. 2020-07-13T06:12:18.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'HVCB2URARHK5PDP7', '--secret', 'iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh', '--endpoints', 'http://smithi014.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:18.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'HVCB2URARHK5PDP7', '--secret', 'iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh', '--endpoints', 'http://smithi014.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:18.434 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key HVCB2URARHK5PDP7 --secret iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh --endpoints http://smithi014.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:12:18.518 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:12:18.518 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:12:18.578 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:12:18.577+0000 7f67e3f600c0 0 failed reading obj info from .rgw.root:zone_info.5fd641b5-9375-4754-8474-97ebd79eafb6: (2) No such file or directory 2020-07-13T06:12:18.579 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:12:18.577+0000 7f67e3f600c0 0 WARNING: could not read zone params for zone id=5fd641b5-9375-4754-8474-97ebd79eafb6 name=test-zone1 2020-07-13T06:12:18.593 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"HVCB2URARHK5PDP7","secret_key":"iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668"} 2020-07-13T06:12:18.594 INFO:teuthology.orchestra.run.smithi014:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2020-07-13T06:12:19.591 INFO:teuthology.orchestra.run.smithi014.stderr:pool 'test-zone2.rgw.buckets.data' created 2020-07-13T06:12:19.604 INFO:teuthology.orchestra.run.smithi014:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2020-07-13T06:12:20.614 INFO:teuthology.orchestra.run.smithi014.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2020-07-13T06:12:20.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:20.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:20.632 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:12:20.667 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:12:20.667 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:12:20.697 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"HVCB2URARHK5PDP7","secret_key":"iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-07-13T06:12:20.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:20.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:20.699 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:12:20.781 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:12:20.782 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:12:22.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2020-07-13T06:12:28.788 INFO:teuthology.orchestra.run.smithi014.stderr:Sending period to new master zone 5fd641b5-9375-4754-8474-97ebd79eafb6 2020-07-13T06:12:28.887 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"587bd98b-69d2-49ba-9dbd-2be72177aade","epoch":2,"predecessor_uuid":"b2a1b5d3-8c44-40d3-9c92-45a62692c41d","sync_status":[],"period_map":{"id":"587bd98b-69d2-49ba-9dbd-2be72177aade","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","zones":[{"id":"5fd641b5-9375-4754-8474-97ebd79eafb6","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","endpoints":["http://smithi014.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5fd641b5-9375-4754-8474-97ebd79eafb6","val":1497020214},{"key":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","val":508986481}]},"master_zonegroup":"test-zonegroup","master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","realm_name":"test-realm","realm_epoch":2} 2020-07-13T06:12:28.888 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2020-07-13T06:12:28.889 INFO:teuthology.orchestra.run:waiting for 300 2020-07-13T06:12:28.930 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:12:28.929+0000 7f430affd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 13453) UID: 0 2020-07-13T06:12:28.930 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:12:28.929+0000 7f432ecf7100 -1 shutting down 2020-07-13T06:12:34.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2020-07-13T06:12:34.991 INFO:tasks.rgw.c2.client.0:Stopped 2020-07-13T06:12:34.992 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-07-13T06:12:34.992 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-07-13T06:12:34.995 INFO:tasks.rgw.c2.client.0:Started 2020-07-13T06:12:34.996 INFO:teuthology.orchestra.run.smithi014:> curl http://smithi014.front.sepia.ceph.com:8000 2020-07-13T06:12:35.012 INFO:teuthology.orchestra.run.smithi014.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-13T06:12:35.012 INFO:teuthology.orchestra.run.smithi014.stderr: Dload Upload Total Spent Left Speed 2020-07-13T06:12:35.016 INFO:teuthology.orchestra.run.smithi014.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi014.front.sepia.ceph.com port 8000: Connection refused 2020-07-13T06:12:35.017 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-07-13T06:12:36.018 INFO:teuthology.orchestra.run.smithi014:> curl http://smithi014.front.sepia.ceph.com:8000 2020-07-13T06:12:36.034 INFO:teuthology.orchestra.run.smithi014.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-13T06:12:36.035 INFO:teuthology.orchestra.run.smithi014.stderr: Dload Upload Total Spent Left Speed 2020-07-13T06:12:36.039 INFO:teuthology.orchestra.run.smithi014.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 53500 2020-07-13T06:12:36.040 INFO:teuthology.orchestra.run.smithi014.stdout:anonymous 2020-07-13T06:12:36.041 INFO:tasks.rgw_multisite:creating zone.. 2020-07-13T06:12:36.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HVCB2URARHK5PDP7', '--secret', 'iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh', '--endpoints', 'http://smithi017.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:36.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HVCB2URARHK5PDP7', '--secret', 'iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh', '--endpoints', 'http://smithi017.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:36.043 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key HVCB2URARHK5PDP7 --secret iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh --endpoints http://smithi017.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:12:36.089 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:12:36.089 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:12:36.164 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:12:36.158+0000 7f055bb820c0 0 failed reading obj info from .rgw.root:zone_info.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e: (2) No such file or directory 2020-07-13T06:12:36.164 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:12:36.158+0000 7f055bb820c0 0 WARNING: could not read zone params for zone id=7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e name=test-zone2 2020-07-13T06:12:36.175 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"HVCB2URARHK5PDP7","secret_key":"iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668"} 2020-07-13T06:12:36.176 INFO:teuthology.orchestra.run.smithi017:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2020-07-13T06:12:37.506 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'test-zone3.rgw.buckets.data' created 2020-07-13T06:12:37.533 INFO:teuthology.orchestra.run.smithi017:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2020-07-13T06:12:38.539 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2020-07-13T06:12:38.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:38.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:38.557 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:12:38.612 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:12:38.613 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:12:38.639 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"HVCB2URARHK5PDP7","secret_key":"iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-07-13T06:12:38.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:38.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:38.641 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:12:38.679 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:12:38.680 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:12:40.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2020-07-13T06:12:46.723 INFO:teuthology.orchestra.run.smithi017.stderr:Sending period to new master zone 5fd641b5-9375-4754-8474-97ebd79eafb6 2020-07-13T06:12:46.817 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"587bd98b-69d2-49ba-9dbd-2be72177aade","epoch":3,"predecessor_uuid":"b2a1b5d3-8c44-40d3-9c92-45a62692c41d","sync_status":[],"period_map":{"id":"587bd98b-69d2-49ba-9dbd-2be72177aade","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","zones":[{"id":"5fd641b5-9375-4754-8474-97ebd79eafb6","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","endpoints":["http://smithi014.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5fd641b5-9375-4754-8474-97ebd79eafb6","val":1497020214},{"key":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","val":508986481},{"key":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","val":617933293}]},"master_zonegroup":"test-zonegroup","master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","realm_name":"test-realm","realm_epoch":2} 2020-07-13T06:12:46.818 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-07-13T06:12:46.819 INFO:teuthology.orchestra.run:waiting for 300 2020-07-13T06:12:46.861 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2020-07-13T06:12:46.858+0000 7fe094f6f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 13519) UID: 0 2020-07-13T06:12:46.862 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2020-07-13T06:12:46.858+0000 7fe0ac646100 -1 shutting down 2020-07-13T06:12:50.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-07-13T06:12:52.921 INFO:tasks.rgw.c1.client.1:Stopped 2020-07-13T06:12:52.922 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-07-13T06:12:52.923 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-07-13T06:12:52.928 INFO:tasks.rgw.c1.client.1:Started 2020-07-13T06:12:52.929 INFO:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2020-07-13T06:12:52.950 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-13T06:12:52.951 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2020-07-13T06:12:52.954 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi017.front.sepia.ceph.com port 8001: Connection refused 2020-07-13T06:12:52.955 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-07-13T06:12:53.956 INFO:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2020-07-13T06:12:53.974 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-13T06:12:53.975 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2020-07-13T06:12:53.979 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2020-07-13T06:12:53.980 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2020-07-13T06:12:53.980 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-07-13T06:12:53.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi017.front.sepia.ceph.com:8000', '--access-key', 'HVCB2URARHK5PDP7', '--secret', 'iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh', '--url', 'http://smithi017.front.sepia.ceph.com:8000', '--access-key', 'HVCB2URARHK5PDP7', '--secret', 'iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:53.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi017.front.sepia.ceph.com:8000', '--access-key', 'HVCB2URARHK5PDP7', '--secret', 'iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh', '--url', 'http://smithi017.front.sepia.ceph.com:8000', '--access-key', 'HVCB2URARHK5PDP7', '--secret', 'iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:53.982 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi017.front.sepia.ceph.com:8000 --access-key HVCB2URARHK5PDP7 --secret iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh --url http://smithi017.front.sepia.ceph.com:8000 --access-key HVCB2URARHK5PDP7 --secret iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:12:54.032 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:12:54.032 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:12:54.076 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:12:54.073+0000 7f5506604700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.3bffea65-3664-4393-a4a6-65c46ad5d668.control err=-107 2020-07-13T06:12:54.100 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"3bffea65-3664-4393-a4a6-65c46ad5d668","name":"test-realm","current_period":"587bd98b-69d2-49ba-9dbd-2be72177aade","epoch":2} 2020-07-13T06:12:54.101 INFO:tasks.rgw_multisite:creating zone.. 2020-07-13T06:12:54.102 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HVCB2URARHK5PDP7', '--secret', 'iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh', '--endpoints', 'http://smithi014.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:54.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HVCB2URARHK5PDP7', '--secret', 'iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh', '--endpoints', 'http://smithi014.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:54.103 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key HVCB2URARHK5PDP7 --secret iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh --endpoints http://smithi014.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:12:54.152 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:12:54.152 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:12:54.205 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:12:54.205+0000 7effe3ae70c0 0 failed reading obj info from .rgw.root:zone_info.5fd641b5-9375-4754-8474-97ebd79eafb6: (2) No such file or directory 2020-07-13T06:12:54.206 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:12:54.205+0000 7effe3ae70c0 0 WARNING: could not read zone params for zone id=5fd641b5-9375-4754-8474-97ebd79eafb6 name=test-zone1 2020-07-13T06:12:54.207 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:12:54.205+0000 7effe3ae70c0 0 failed reading obj info from .rgw.root:zone_info.e66a1e9b-6861-42ff-8eed-7287194f3fe0: (2) No such file or directory 2020-07-13T06:12:54.208 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:12:54.205+0000 7effe3ae70c0 0 WARNING: could not read zone params for zone id=e66a1e9b-6861-42ff-8eed-7287194f3fe0 name=test-zone3 2020-07-13T06:12:54.223 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"HVCB2URARHK5PDP7","secret_key":"iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668"} 2020-07-13T06:12:54.224 INFO:teuthology.orchestra.run.smithi014:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2020-07-13T06:12:55.491 INFO:teuthology.orchestra.run.smithi014.stderr:pool 'test-zone4.rgw.buckets.data' created 2020-07-13T06:12:55.513 INFO:teuthology.orchestra.run.smithi014:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2020-07-13T06:12:56.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-07-13T06:12:56.510 INFO:teuthology.orchestra.run.smithi014.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2020-07-13T06:12:56.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:56.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:56.535 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:12:56.581 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:12:56.582 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:12:56.608 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"HVCB2URARHK5PDP7","secret_key":"iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":""} 2020-07-13T06:12:56.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:56.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:12:56.609 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:12:56.650 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:12:56.650 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:13:04.700 INFO:teuthology.orchestra.run.smithi014.stderr:Sending period to new master zone 5fd641b5-9375-4754-8474-97ebd79eafb6 2020-07-13T06:13:04.959 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"587bd98b-69d2-49ba-9dbd-2be72177aade","epoch":4,"predecessor_uuid":"b2a1b5d3-8c44-40d3-9c92-45a62692c41d","sync_status":[],"period_map":{"id":"587bd98b-69d2-49ba-9dbd-2be72177aade","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","zones":[{"id":"5fd641b5-9375-4754-8474-97ebd79eafb6","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","endpoints":["http://smithi014.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","name":"test-zone4","endpoints":["http://smithi014.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5fd641b5-9375-4754-8474-97ebd79eafb6","val":1497020214},{"key":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","val":508986481},{"key":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","val":2882337890},{"key":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","val":617933293}]},"master_zonegroup":"test-zonegroup","master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","realm_name":"test-realm","realm_epoch":2} 2020-07-13T06:13:04.960 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2020-07-13T06:13:04.960 INFO:teuthology.orchestra.run:waiting for 300 2020-07-13T06:13:05.000 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:13:05.001+0000 7fc3e96ba700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 13456) UID: 0 2020-07-13T06:13:05.001 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:13:05.001+0000 7fc400d91100 -1 shutting down 2020-07-13T06:13:08.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2020-07-13T06:13:11.062 INFO:tasks.rgw.c2.client.1:Stopped 2020-07-13T06:13:11.063 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-07-13T06:13:11.063 INFO:teuthology.orchestra.run.smithi014:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-07-13T06:13:11.066 INFO:tasks.rgw.c2.client.1:Started 2020-07-13T06:13:11.067 INFO:teuthology.orchestra.run.smithi014:> curl http://smithi014.front.sepia.ceph.com:8001 2020-07-13T06:13:11.083 INFO:teuthology.orchestra.run.smithi014.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-13T06:13:11.083 INFO:teuthology.orchestra.run.smithi014.stderr: Dload Upload Total Spent Left Speed 2020-07-13T06:13:11.087 INFO:teuthology.orchestra.run.smithi014.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi014.front.sepia.ceph.com port 8001: Connection refused 2020-07-13T06:13:11.088 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-07-13T06:13:12.090 INFO:teuthology.orchestra.run.smithi014:> curl http://smithi014.front.sepia.ceph.com:8001 2020-07-13T06:13:12.104 INFO:teuthology.orchestra.run.smithi014.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-13T06:13:12.105 INFO:teuthology.orchestra.run.smithi014.stderr: Dload Upload Total Spent Left Speed 2020-07-13T06:13:12.109 INFO:teuthology.orchestra.run.smithi014.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 33500 0 --:--:-- --:--:-- --:--:-- 33500 2020-07-13T06:13:12.110 INFO:teuthology.orchestra.run.smithi014.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-005f0bfb78-115a-test-zone4","HostId":"115a-test-zone4-test-zonegroup"} 2020-07-13T06:13:12.110 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2020-07-13T06:13:12.111 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2020-07-13T06:13:12.312 INFO:tasks.rgw_multisite_tests:creating test user.. 2020-07-13T06:13:12.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:13:12.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:13:12.313 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:13:12.357 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:13:12.358 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:13:12.574 INFO:teuthology.orchestra.run.smithi017.stdout:{"user_id":"rgw-multisite-test-user","display_name":"Multisite Test User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"S591ZLYFQ3X6GEY1WWRG","secret_key":"OMUqSvjYXG0v26FUXSGXeTr8ASSdxKkZb4TdOPoR"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2020-07-13T06:13:12.596 INFO:rgw_multi.tests:meta checkpoint 2020-07-13T06:13:12.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:12.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:12.599 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:13:12.679 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:13:12.680 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:13:12.870 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=587bd98b-69d2-49ba-9dbd-2be72177aade 2020-07-13T06:13:12.908 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594620792.569571_63.1","last_update":"2020-07-13T06:13:12.569571Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:13:12.909 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:13:12.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:12.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:12.910 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:13:12.951 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:13:12.952 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:13:13.172 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:13:13.172 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:13:14.280 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2020-07-13T06:13:18.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:18.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:18.175 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:13:18.215 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:13:18.215 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:13:18.453 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:13:18.454 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:13:23.455 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:23.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:23.457 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:13:23.500 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:13:23.500 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:13:23.731 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:13:23.732 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:13:28.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:28.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:28.735 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:13:28.775 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:13:28.776 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:13:29.014 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:13:29.020 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:13:34.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:34.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:34.023 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:13:34.066 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:13:34.066 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:13:34.302 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:13:34.303 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:13:39.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:39.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:39.306 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:13:39.347 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:13:39.348 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:13:39.592 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:13:39.593 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:13:44.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:44.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:44.596 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:13:44.639 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:13:44.640 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:13:44.863 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:13:44.864 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:13:49.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:49.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:49.867 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:13:49.908 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:13:49.908 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:13:50.123 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:13:50.123 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:13:55.125 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:55.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:13:55.126 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:13:55.169 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:13:55.169 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:13:55.400 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:13:55.401 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:14:00.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:00.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:00.405 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:14:00.445 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:14:00.446 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:14:00.678 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:14:00.679 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:14:05.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:05.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:05.682 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:14:05.724 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:14:05.725 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:14:05.955 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:14:05.955 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:14:10.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:10.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:10.959 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:14:11.000 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:14:11.001 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:14:11.231 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:14:11.232 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:14:16.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:16.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:16.235 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:14:16.276 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:14:16.285 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:14:16.510 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:14:16.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:14:21.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:21.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:21.515 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:14:21.556 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:14:21.557 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:14:21.797 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:14:21.798 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:14:26.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:26.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:26.804 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:14:26.847 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:14:26.848 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:14:27.076 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:14:27.077 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:14:32.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:32.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:32.080 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:14:32.124 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:14:32.125 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:14:32.359 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:14:32.360 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:14:37.362 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:37.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:37.363 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:14:37.405 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:14:37.406 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:14:37.634 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:14:37.635 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:14:42.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:42.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:42.638 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:14:42.680 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:14:42.680 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:14:42.932 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:14:42.933 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:14:47.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:47.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:47.936 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:14:47.976 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:14:47.976 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:14:48.222 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:14:48.222 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:14:53.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:53.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:53.226 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:14:53.267 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:14:53.268 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:14:53.500 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:14:53.501 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:14:58.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:58.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:14:58.515 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:14:58.556 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:14:58.556 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:14:58.798 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:14:58.799 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:15:03.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:03.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:03.803 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:03.847 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:03.848 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:04.095 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:04.096 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594620792.569571_63.1 target= 2020-07-13T06:15:09.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:09.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:09.102 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:09.145 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:09.146 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:09.390 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:09.391 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:15:09.392 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:15:09.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:09.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:09.393 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:09.436 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:09.436 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:09.690 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:09.694 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:15:09.695 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:15:09.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:09.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:09.697 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:09.739 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:09.740 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:10.010 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:10.011 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:15:10.012 INFO:rgw_multi.tests:meta checkpoint 2020-07-13T06:15:10.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:10.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:10.013 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:10.060 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:10.060 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:10.259 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=587bd98b-69d2-49ba-9dbd-2be72177aade 2020-07-13T06:15:10.295 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594620792.569571_63.1","last_update":"2020-07-13T06:13:12.569571Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:15:10.296 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:15:10.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:10.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:10.297 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:10.345 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:10.346 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:10.575 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:10.576 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:15:10.576 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:15:10.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:10.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:10.577 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:10.616 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:10.617 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:10.853 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:10.854 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:15:10.854 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:15:10.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:10.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:10.855 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:10.902 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:10.902 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:11.173 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:11.174 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:15:11.174 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2020-07-13T06:15:11.189 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-1 2020-07-13T06:15:14.581 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-2 2020-07-13T06:15:18.593 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-3 2020-07-13T06:15:18.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:18.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:18.655 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:18.691 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:18.692 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:18.889 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=587bd98b-69d2-49ba-9dbd-2be72177aade 2020-07-13T06:15:18.932 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594620792.569571_63.1","last_update":"2020-07-13T06:13:12.569571Z"},{"marker":"1_1594620918.629897_78.1","last_update":"2020-07-13T06:15:18.629897Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:15:18.933 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:15:18.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:18.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:18.938 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:18.980 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:18.981 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:19.249 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620918.629897_78.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:18.629897Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:19.250 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:15:19.250 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:15:19.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:19.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:19.252 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:19.292 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:19.292 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:19.538 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620918.629897_78.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:18.629897Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:19.539 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:15:19.540 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:15:19.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:19.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:19.542 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:19.589 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:19.589 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:19.849 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620918.629897_78.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:18.629897Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:19.850 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:15:21.788 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2020-07-13T06:15:21.790 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-4 2020-07-13T06:15:21.883 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-5 2020-07-13T06:15:21.945 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-6 2020-07-13T06:15:22.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:22.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:22.016 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:22.055 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:22.055 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:22.238 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=587bd98b-69d2-49ba-9dbd-2be72177aade 2020-07-13T06:15:22.269 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594620792.569571_63.1","last_update":"2020-07-13T06:13:12.569571Z"},{"marker":"1_1594620921.991312_90.1","last_update":"2020-07-13T06:15:21.991312Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:15:22.270 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:15:22.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:22.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:22.273 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:22.316 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:22.317 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:22.553 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620921.991312_90.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:21.991312Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:22.553 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:15:22.554 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:15:22.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:22.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:22.556 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:22.601 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:22.602 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:22.827 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620921.991312_90.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:21.991312Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:22.828 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:15:22.828 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:15:22.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:22.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:22.830 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:22.870 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:22.871 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:23.144 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620921.991312_90.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:21.991312Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:23.145 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:15:26.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:26.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:26.959 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:26.994 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:26.994 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:27.187 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=587bd98b-69d2-49ba-9dbd-2be72177aade 2020-07-13T06:15:27.226 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594620792.569571_63.1","last_update":"2020-07-13T06:13:12.569571Z"},{"marker":"1_1594620926.570739_126.1","last_update":"2020-07-13T06:15:26.570739Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:15:27.226 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:15:27.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:27.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:27.228 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:27.270 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:27.270 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:27.505 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620926.570739_126.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:26.570739Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:27.505 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:15:27.506 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:15:27.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:27.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:27.508 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:27.544 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:27.544 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:27.761 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620926.570739_126.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:26.570739Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:27.762 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:15:27.763 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:15:27.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:27.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:27.765 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:27.807 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:27.808 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:28.079 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620926.570739_126.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:26.570739Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:28.080 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:15:28.978 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2020-07-13T06:15:28.979 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-7 2020-07-13T06:15:29.016 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-8 2020-07-13T06:15:29.072 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-9 2020-07-13T06:15:29.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:29.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:29.118 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:29.160 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:29.160 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:29.337 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=587bd98b-69d2-49ba-9dbd-2be72177aade 2020-07-13T06:15:29.375 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594620792.569571_63.1","last_update":"2020-07-13T06:13:12.569571Z"},{"marker":"1_1594620929.100492_138.1","last_update":"2020-07-13T06:15:29.100492Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:15:29.376 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:15:29.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:29.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:29.379 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:29.418 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:29.419 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:29.659 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620929.100492_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:29.100492Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:29.659 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:15:29.660 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:15:29.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:29.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:29.662 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:29.705 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:29.705 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:29.930 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620929.100492_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:29.100492Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:29.932 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:15:29.932 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:15:29.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:29.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:29.934 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:29.976 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:29.977 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:30.240 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620929.100492_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:29.100492Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:30.240 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:15:33.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:33.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:33.475 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:33.510 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:33.510 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:33.692 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=587bd98b-69d2-49ba-9dbd-2be72177aade 2020-07-13T06:15:33.730 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594620792.569571_63.1","last_update":"2020-07-13T06:13:12.569571Z"},{"marker":"1_1594620933.443793_144.1","last_update":"2020-07-13T06:15:33.443793Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:15:33.731 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:15:33.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:33.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:33.733 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:33.777 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:33.777 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:34.014 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:34.015 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:15:34.016 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:15:34.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:34.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:34.018 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:34.060 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:34.061 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:34.280 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:34.281 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:15:34.282 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:15:34.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:34.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:34.283 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:34.323 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:34.323 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:34.580 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620792.569571_63.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:13:12.569571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:34.581 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:15:35.332 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2020-07-13T06:15:35.333 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-10 2020-07-13T06:15:35.366 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-11 2020-07-13T06:15:35.420 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-12 2020-07-13T06:15:36.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:36.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:36.081 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:36.120 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:36.121 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:36.369 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=587bd98b-69d2-49ba-9dbd-2be72177aade 2020-07-13T06:15:36.401 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594620935.457356_94.1","last_update":"2020-07-13T06:15:35.457356Z"},{"marker":"1_1594620933.443793_144.1","last_update":"2020-07-13T06:15:33.443793Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:15:36.402 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:15:36.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:36.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:36.403 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:36.449 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:36.449 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:36.672 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620935.457356_94.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:15:35.457356Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:36.673 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:15:36.673 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:15:36.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:36.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:36.675 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:36.716 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:36.717 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:36.944 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620935.457356_94.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:15:35.457356Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:36.945 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:15:36.945 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:15:36.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:36.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:36.947 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:36.990 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:36.990 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:37.255 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620935.457356_94.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:15:35.457356Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:37.256 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:15:37.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:37.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:37.259 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:37.300 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:37.301 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:37.526 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.11.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.14.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.6.5"},{"key":8,"val":"00000000001.15.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:15:37.527 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-10 2020-07-13T06:15:37.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:37.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:37.528 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:37.575 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:37.575 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:37.814 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.11.5","timestamp":"2020-07-13T06:15:35.646003Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.14.5","timestamp":"2020-07-13T06:15:35.642263Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5","timestamp":"2020-07-13T06:15:35.878058Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.5","timestamp":"2020-07-13T06:15:35.878081Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:15:37.814 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-10 2020-07-13T06:15:37.815 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-10 zones={test-zone2, test-zone1} 2020-07-13T06:15:38.182 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:15:38.197 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-13T06:15:38.198 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-13T06:15:38.198 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-13T06:15:38.199 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:15:38.199 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:15:38.210 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-13T06:15:38.210 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-13T06:15:38.211 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-13T06:15:38.211 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:15:38.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-13T06:15:38.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-13T06:15:38.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-13T06:15:38.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-13T06:15:38.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-13T06:15:38.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-13T06:15:38.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-13T06:15:38.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-13T06:15:38.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-13T06:15:39.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:39.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:39.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:39.372 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-10 zones={test-zone2, test-zone1} 2020-07-13T06:15:39.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:39.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:39.374 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:39.411 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:39.412 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:39.616 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.11.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.14.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.6.5"},{"key":8,"val":"00000000001.15.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:15:39.618 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-10 2020-07-13T06:15:39.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:39.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:39.621 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:39.702 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:39.703 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:39.948 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.11.5","timestamp":"2020-07-13T06:15:35.637327Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.14.5","timestamp":"2020-07-13T06:15:35.637392Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5","timestamp":"2020-07-13T06:15:35.873536Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.5","timestamp":"2020-07-13T06:15:35.873629Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:15:39.949 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-10 2020-07-13T06:15:39.950 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-10 zones={test-zone3, test-zone1} 2020-07-13T06:15:41.173 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:15:41.183 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-13T06:15:41.184 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-13T06:15:41.184 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-13T06:15:41.185 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:15:41.185 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:15:41.191 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-13T06:15:41.191 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-13T06:15:41.192 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-13T06:15:41.192 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:15:41.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-13T06:15:41.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-13T06:15:41.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-13T06:15:41.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-13T06:15:41.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-13T06:15:41.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-13T06:15:41.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-13T06:15:41.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-13T06:15:41.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-13T06:15:42.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:42.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:42.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:42.314 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-10 zones={test-zone3, test-zone1} 2020-07-13T06:15:42.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:42.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:42.316 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:42.353 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:42.353 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:42.570 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.11.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.14.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.6.5"},{"key":8,"val":"00000000001.15.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:15:42.571 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-10 2020-07-13T06:15:42.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:42.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:42.572 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:42.619 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:42.619 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:42.995 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.11.5","timestamp":"2020-07-13T06:15:35.483410Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.14.5","timestamp":"2020-07-13T06:15:35.533717Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.6.5","timestamp":"2020-07-13T06:15:35.626775Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.15.5","timestamp":"2020-07-13T06:15:35.581137Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:15:42.996 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-10 2020-07-13T06:15:42.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:42.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:42.998 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:43.041 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:43.041 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:43.276 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.6.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.18.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.15.5"},{"key":8,"val":"00000000001.13.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:15:43.277 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-11 2020-07-13T06:15:43.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:43.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:43.279 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:43.324 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:43.324 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:43.562 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5","timestamp":"2020-07-13T06:15:35.886759Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.18.5","timestamp":"2020-07-13T06:15:35.929276Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.5","timestamp":"2020-07-13T06:15:35.888702Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-07-13T06:15:35.934697Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:15:43.563 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-11 2020-07-13T06:15:43.564 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-11 zones={test-zone1, test-zone2} 2020-07-13T06:15:43.947 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:15:43.959 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-13T06:15:43.961 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-13T06:15:43.962 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-13T06:15:43.963 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:15:43.964 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:15:43.973 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-13T06:15:43.974 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-13T06:15:43.975 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-13T06:15:43.976 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:15:43.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-13T06:15:44.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-13T06:15:44.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-13T06:15:44.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-13T06:15:44.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-13T06:15:44.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-13T06:15:44.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-13T06:15:44.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-13T06:15:44.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-13T06:15:44.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:44.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:45.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:45.159 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-11 zones={test-zone1, test-zone2} 2020-07-13T06:15:45.160 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:45.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:45.163 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:45.202 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:45.202 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:45.425 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.6.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.18.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.15.5"},{"key":8,"val":"00000000001.13.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:15:45.425 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-11 2020-07-13T06:15:45.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:45.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:45.428 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:45.473 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:45.473 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:45.717 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5","timestamp":"2020-07-13T06:15:35.925296Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.18.5","timestamp":"2020-07-13T06:15:35.886708Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.5","timestamp":"2020-07-13T06:15:35.925349Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-07-13T06:15:35.929378Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:15:45.718 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-11 2020-07-13T06:15:45.718 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-11 zones={test-zone3, test-zone2} 2020-07-13T06:15:46.673 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:15:46.685 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-13T06:15:46.686 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-13T06:15:46.687 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-13T06:15:46.687 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:15:46.688 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:15:46.696 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-13T06:15:46.697 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-13T06:15:46.697 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-13T06:15:46.698 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:15:46.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-13T06:15:46.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-13T06:15:46.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-13T06:15:47.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-13T06:15:47.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-13T06:15:47.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-13T06:15:47.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-13T06:15:47.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-13T06:15:47.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-13T06:15:47.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:47.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:47.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:47.882 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-11 zones={test-zone3, test-zone2} 2020-07-13T06:15:47.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:47.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:47.885 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:47.930 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:47.930 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:48.180 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.9.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.14.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.7.5"},{"key":8,"val":"00000000001.16.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:15:48.181 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-12 2020-07-13T06:15:48.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:48.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:48.183 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:48.232 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:48.232 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:48.466 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5","timestamp":"2020-07-13T06:15:35.993206Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.14.5","timestamp":"2020-07-13T06:15:36.193149Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5","timestamp":"2020-07-13T06:15:36.154577Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.5","timestamp":"2020-07-13T06:15:36.154690Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:15:48.467 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-12 2020-07-13T06:15:48.467 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-12 zones={test-zone1, test-zone3} 2020-07-13T06:15:48.641 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:15:48.649 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-13T06:15:48.649 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-13T06:15:48.650 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-13T06:15:48.650 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:15:48.651 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:15:48.658 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-13T06:15:48.659 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-13T06:15:48.659 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-13T06:15:48.660 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:15:48.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-13T06:15:48.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-13T06:15:48.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-13T06:15:48.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-13T06:15:49.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-13T06:15:49.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-13T06:15:49.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-13T06:15:49.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-13T06:15:49.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-13T06:15:49.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:49.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:49.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:49.819 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-12 zones={test-zone1, test-zone3} 2020-07-13T06:15:49.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:49.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:49.821 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:49.861 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:49.861 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:50.081 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.9.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.14.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.7.5"},{"key":8,"val":"00000000001.16.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:15:50.082 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-12 2020-07-13T06:15:50.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:50.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:50.084 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-12 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:50.130 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:50.130 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:50.402 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5","timestamp":"2020-07-13T06:15:36.002283Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.14.5","timestamp":"2020-07-13T06:15:36.194131Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5","timestamp":"2020-07-13T06:15:36.190237Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.5","timestamp":"2020-07-13T06:15:36.194162Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:15:50.403 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-12 2020-07-13T06:15:50.404 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-12 zones={test-zone2, test-zone3} 2020-07-13T06:15:52.137 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:15:52.149 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-13T06:15:52.150 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-13T06:15:52.150 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-13T06:15:52.151 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:15:52.151 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:15:52.163 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-13T06:15:52.163 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-13T06:15:52.164 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-13T06:15:52.164 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:15:52.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-13T06:15:52.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-13T06:15:52.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-13T06:15:52.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-13T06:15:52.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-13T06:15:52.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-13T06:15:52.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-13T06:15:52.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-13T06:15:52.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-13T06:15:53.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:53.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:53.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:53.340 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-12 zones={test-zone2, test-zone3} 2020-07-13T06:15:53.341 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2020-07-13T06:15:53.342 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-13 2020-07-13T06:15:53.579 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-14 2020-07-13T06:15:53.640 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-15 2020-07-13T06:15:53.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:53.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:53.852 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:53.890 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:53.891 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:54.076 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=587bd98b-69d2-49ba-9dbd-2be72177aade 2020-07-13T06:15:54.105 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594620953.682997_125.1","last_update":"2020-07-13T06:15:53.682997Z"},{"marker":"1_1594620933.443793_144.1","last_update":"2020-07-13T06:15:33.443793Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:15:54.106 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:15:54.106 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:54.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:54.108 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:54.147 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:54.147 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:54.367 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620953.682997_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:15:53.682997Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:54.368 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:15:54.369 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:15:54.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:54.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:54.371 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:54.411 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:54.411 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:54.611 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620953.682997_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:15:53.682997Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:54.612 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:15:54.613 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:15:54.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:54.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:54.615 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:54.663 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:54.663 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:54.928 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620953.682997_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:15:53.682997Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:15:54.929 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:15:54.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:54.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:54.939 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:54.982 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:54.983 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:55.198 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.13.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:15:55.199 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-13 2020-07-13T06:15:55.199 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:55.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:55.200 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:55.247 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:55.247 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:55.491 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-07-13T06:15:53.850114Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:15:55.491 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-13 2020-07-13T06:15:55.492 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-13 zones={test-zone2, test-zone1} 2020-07-13T06:15:55.935 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:15:55.946 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:15:55.947 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:15:55.998 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:15:56.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:56.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:56.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:56.345 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-13 zones={test-zone2, test-zone1} 2020-07-13T06:15:56.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:56.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:56.347 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:56.387 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:56.387 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:56.595 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.13.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:15:56.595 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-13 2020-07-13T06:15:56.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:56.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:56.597 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:56.681 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:56.681 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:56.923 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-07-13T06:15:53.849216Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:15:56.924 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-13 2020-07-13T06:15:56.924 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-13 zones={test-zone3, test-zone1} 2020-07-13T06:15:58.428 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:15:58.437 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:15:58.438 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:15:58.490 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:15:58.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:58.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:58.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:15:58.844 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-13 zones={test-zone3, test-zone1} 2020-07-13T06:15:58.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:58.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:58.846 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:58.886 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:58.886 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:59.106 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.13.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:15:59.106 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-13 2020-07-13T06:15:59.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:59.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:59.108 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:59.162 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:59.162 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:59.428 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-07-13T06:15:53.713507Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:15:59.428 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-13 2020-07-13T06:15:59.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:59.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:59.430 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:59.473 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:59.474 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:15:59.716 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.8.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:15:59.716 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-14 2020-07-13T06:15:59.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:59.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:15:59.718 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:15:59.764 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:15:59.764 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:00.006 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5","timestamp":"2020-07-13T06:15:53.925254Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:00.007 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-14 2020-07-13T06:16:00.008 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-14 zones={test-zone1, test-zone2} 2020-07-13T06:16:00.428 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:16:00.441 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:16:00.442 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:16:00.492 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:16:00.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:16:00.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:16:00.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:16:00.850 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-14 zones={test-zone1, test-zone2} 2020-07-13T06:16:00.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:00.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:00.851 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:00.890 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:00.891 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:01.110 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.8.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:16:01.110 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-14 2020-07-13T06:16:01.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:01.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:01.112 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:01.157 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:01.158 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:01.400 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5","timestamp":"2020-07-13T06:15:53.921230Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:01.401 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-14 2020-07-13T06:16:01.402 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-14 zones={test-zone3, test-zone2} 2020-07-13T06:16:01.739 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:16:01.790 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:16:01.791 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:16:01.810 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:16:01.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:16:01.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:16:02.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:16:02.330 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-14 zones={test-zone3, test-zone2} 2020-07-13T06:16:02.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:02.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:02.332 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:02.378 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:02.379 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:02.583 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.5.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:16:02.584 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-15 2020-07-13T06:16:02.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:02.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:02.585 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:02.669 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:02.669 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:02.924 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.5.5","timestamp":"2020-07-13T06:15:53.997181Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:02.925 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-15 2020-07-13T06:16:02.926 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-15 zones={test-zone1, test-zone3} 2020-07-13T06:16:03.095 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:16:03.147 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:16:03.148 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:16:03.158 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:16:03.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:16:03.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:16:03.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:16:03.519 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-15 zones={test-zone1, test-zone3} 2020-07-13T06:16:03.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:03.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:03.521 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:03.561 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:03.561 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:03.796 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.5.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:16:03.797 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-15 2020-07-13T06:16:03.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:03.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:03.800 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:03.845 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:03.846 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:04.085 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.5.5","timestamp":"2020-07-13T06:15:53.998030Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:04.086 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-15 2020-07-13T06:16:04.087 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-15 zones={test-zone2, test-zone3} 2020-07-13T06:16:05.410 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:16:05.423 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:16:05.426 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:16:05.478 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-13T06:16:05.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:16:05.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:16:05.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-13T06:16:05.841 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-15 zones={test-zone2, test-zone3} 2020-07-13T06:16:05.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:05.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:05.893 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:05.932 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:05.933 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:06.143 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.15.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:16:06.143 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-13 2020-07-13T06:16:06.144 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:06.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:06.146 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:06.194 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:06.194 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:06.426 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-07-13T06:15:53.850114Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:06.427 WARNING:rgw_multi.tests:bucket cngrzz-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.15.5 target=00000000001.13.5 2020-07-13T06:16:11.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:11.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:11.430 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:11.476 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:11.477 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:11.691 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-07-13T06:15:53.850114Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:11.692 WARNING:rgw_multi.tests:bucket cngrzz-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.15.5 target=00000000001.13.5 2020-07-13T06:16:16.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:16.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:16.696 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:16.741 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:16.742 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:16.977 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-07-13T06:15:53.850114Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:16.978 WARNING:rgw_multi.tests:bucket cngrzz-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.15.5 target=00000000001.13.5 2020-07-13T06:16:21.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:21.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:21.981 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:22.029 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:22.029 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:22.271 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-07-13T06:15:53.850114Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:22.271 WARNING:rgw_multi.tests:bucket cngrzz-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.15.5 target=00000000001.13.5 2020-07-13T06:16:27.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:27.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:27.275 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:27.320 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:27.321 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:27.560 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.15.5","timestamp":"2020-07-13T06:15:53.713507Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:27.561 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-13 2020-07-13T06:16:27.561 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-13 zones={test-zone2, test-zone1} 2020-07-13T06:16:28.699 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:16:28.707 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:16:28.822 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-13 zones={test-zone2, test-zone1} 2020-07-13T06:16:28.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:28.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:28.824 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:28.867 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:28.867 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:29.091 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.15.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:16:29.092 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-13 2020-07-13T06:16:29.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:29.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:29.094 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:29.180 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:29.181 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:29.410 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.15.5","timestamp":"2020-07-13T06:15:53.713507Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:29.411 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-13 2020-07-13T06:16:29.411 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-13 zones={test-zone3, test-zone1} 2020-07-13T06:16:30.833 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:16:30.842 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:16:30.950 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-13 zones={test-zone3, test-zone1} 2020-07-13T06:16:30.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:30.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:30.952 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:30.994 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:30.995 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:31.220 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.15.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:16:31.221 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-13 2020-07-13T06:16:31.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:31.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:31.223 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:31.266 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:31.267 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:31.534 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.15.5","timestamp":"2020-07-13T06:15:53.713507Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:31.535 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-13 2020-07-13T06:16:31.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:31.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:31.676 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:31.719 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:31.719 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:31.929 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.11.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:16:31.929 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-14 2020-07-13T06:16:31.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:31.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:31.931 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:31.975 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:31.975 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:32.216 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.11.5","timestamp":"2020-07-13T06:15:53.762299Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:32.217 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-14 2020-07-13T06:16:32.218 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-14 zones={test-zone1, test-zone2} 2020-07-13T06:16:32.246 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:16:32.294 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:16:32.366 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-14 zones={test-zone1, test-zone2} 2020-07-13T06:16:32.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:32.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:32.368 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:32.408 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:32.409 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:32.635 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.11.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:16:32.635 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-14 2020-07-13T06:16:32.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:32.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:32.637 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:32.681 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:32.682 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:32.901 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.11.5","timestamp":"2020-07-13T06:15:53.762299Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:32.902 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-14 2020-07-13T06:16:32.902 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-14 zones={test-zone3, test-zone2} 2020-07-13T06:16:32.969 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:16:33.040 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:16:33.143 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-14 zones={test-zone3, test-zone2} 2020-07-13T06:16:33.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:33.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:33.182 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:33.229 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:33.230 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:33.440 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.8.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:16:33.440 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-15 2020-07-13T06:16:33.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:33.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:33.442 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:33.531 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:33.532 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:33.731 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.8.5","timestamp":"2020-07-13T06:15:53.811184Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:33.732 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-15 2020-07-13T06:16:33.732 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-15 zones={test-zone1, test-zone3} 2020-07-13T06:16:34.038 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:16:34.090 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:16:34.164 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-15 zones={test-zone1, test-zone3} 2020-07-13T06:16:34.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:34.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:34.167 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:34.211 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:34.212 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:34.426 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.8.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:16:34.427 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-15 2020-07-13T06:16:34.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:34.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:34.429 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:34.473 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:34.474 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:34.709 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.8.5","timestamp":"2020-07-13T06:15:53.811184Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:34.710 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-15 2020-07-13T06:16:34.711 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-15 zones={test-zone2, test-zone3} 2020-07-13T06:16:35.639 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:16:35.649 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:16:35.767 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-15 zones={test-zone2, test-zone3} 2020-07-13T06:16:35.768 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2020-07-13T06:16:35.769 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-16 2020-07-13T06:16:35.914 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-17 2020-07-13T06:16:35.967 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-18 2020-07-13T06:16:36.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:36.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:36.075 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:36.108 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:36.108 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:36.283 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=587bd98b-69d2-49ba-9dbd-2be72177aade 2020-07-13T06:16:36.313 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594620996.066494_203.1","last_update":"2020-07-13T06:16:36.066494Z"},{"marker":"1_1594620933.443793_144.1","last_update":"2020-07-13T06:15:33.443793Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:16:36.314 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:16:36.315 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:36.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:36.316 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:36.365 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:36.365 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:36.582 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620996.066494_203.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:16:36.066494Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:16:36.583 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:16:36.584 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:16:36.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:36.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:36.585 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:36.625 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:36.625 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:36.842 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620996.066494_203.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:16:36.066494Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:16:36.843 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:16:36.843 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:16:36.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:36.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:36.846 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:36.886 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:36.886 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:37.163 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594620996.066494_203.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:16:36.066494Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:16:37.163 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:16:38.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:38.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:38.188 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:38.231 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:38.231 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:38.426 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.15.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:16:38.427 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-16 2020-07-13T06:16:38.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:38.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:38.429 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:38.474 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:38.475 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:38.698 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000002.15.10","timestamp":"2020-07-13T06:16:38.274123Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:38.699 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-16 2020-07-13T06:16:38.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:38.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:38.701 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:38.739 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:38.740 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:38.940 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:16:38.941 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-16 2020-07-13T06:16:38.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:38.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:38.943 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:39.031 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:39.035 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:39.266 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000002.15.10","timestamp":"2020-07-13T06:16:38.269222Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:39.267 WARNING:rgw_multi.tests:bucket cngrzz-16 zone test-zone3 behind zone test-zone1: shard=5 master=00000000005.21.9 target=00000000002.15.10 2020-07-13T06:16:44.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:44.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:44.270 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:44.317 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:44.317 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:44.543 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000002.15.10","timestamp":"2020-07-13T06:16:38.269222Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:44.544 WARNING:rgw_multi.tests:bucket cngrzz-16 zone test-zone3 behind zone test-zone1: shard=5 master=00000000005.21.9 target=00000000002.15.10 2020-07-13T06:16:49.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:49.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:49.547 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:49.587 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:49.588 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:49.819 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000005.21.9","timestamp":"2020-07-13T06:16:38.148200Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:49.819 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-16 2020-07-13T06:16:49.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:49.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:49.821 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:49.902 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:49.903 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:50.112 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000006.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:16:50.112 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-16 2020-07-13T06:16:50.113 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:50.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:50.114 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:50.159 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:50.159 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:50.430 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.21.9","timestamp":"2020-07-13T06:16:38.148200Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:50.431 WARNING:rgw_multi.tests:bucket cngrzz-16 zone test-zone4 behind zone test-zone1: shard=5 master=00000000006.23.9 target=00000000005.21.9 2020-07-13T06:16:55.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:55.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:16:55.434 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:16:55.478 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:16:55.479 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:16:55.754 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.21.9","timestamp":"2020-07-13T06:16:38.148200Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:16:55.755 WARNING:rgw_multi.tests:bucket cngrzz-16 zone test-zone4 behind zone test-zone1: shard=5 master=00000000006.23.9 target=00000000005.21.9 2020-07-13T06:17:00.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:00.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:00.758 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:00.802 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:00.803 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:01.075 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.21.9","timestamp":"2020-07-13T06:16:38.148200Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:17:01.075 WARNING:rgw_multi.tests:bucket cngrzz-16 zone test-zone4 behind zone test-zone1: shard=5 master=00000000006.23.9 target=00000000005.21.9 2020-07-13T06:17:06.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:06.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:06.079 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:06.125 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:06.125 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:06.402 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.21.9","timestamp":"2020-07-13T06:16:38.148200Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:17:06.403 WARNING:rgw_multi.tests:bucket cngrzz-16 zone test-zone4 behind zone test-zone1: shard=5 master=00000000006.23.9 target=00000000005.21.9 2020-07-13T06:17:11.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:11.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:11.406 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:11.452 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:11.452 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:11.715 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.21.9","timestamp":"2020-07-13T06:16:38.148200Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:17:11.716 WARNING:rgw_multi.tests:bucket cngrzz-16 zone test-zone4 behind zone test-zone1: shard=5 master=00000000006.23.9 target=00000000005.21.9 2020-07-13T06:17:16.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:16.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:16.719 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:16.765 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:16.766 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:17.038 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-07-13T06:16:38.148200Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:17:17.039 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-16 2020-07-13T06:17:17.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:17.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:17.284 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:17.326 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:17.326 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:17.533 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.19.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:17:17.534 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-17 2020-07-13T06:17:17.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:17.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:17.536 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:17.578 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:17.578 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:17.780 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000002.19.10","timestamp":"2020-07-13T06:17:17.373355Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:17:17.780 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-17 2020-07-13T06:17:17.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:17.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:17.782 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:17.826 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:17.826 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:18.021 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000006.27.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:17:18.022 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-17 2020-07-13T06:17:18.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:18.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:18.023 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:18.068 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:18.068 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:18.292 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000002.19.10","timestamp":"2020-07-13T06:17:17.373363Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:17:18.293 WARNING:rgw_multi.tests:bucket cngrzz-17 zone test-zone3 behind zone test-zone2: shard=5 master=00000000006.27.9 target=00000000002.19.10 2020-07-13T06:17:23.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:23.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:23.296 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:23.342 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:23.342 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:23.578 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000002.19.10","timestamp":"2020-07-13T06:17:17.373363Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:17:23.579 WARNING:rgw_multi.tests:bucket cngrzz-17 zone test-zone3 behind zone test-zone2: shard=5 master=00000000006.27.9 target=00000000002.19.10 2020-07-13T06:17:28.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:28.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:28.581 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:28.621 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:28.622 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:28.864 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000002.19.10","timestamp":"2020-07-13T06:17:17.373363Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:17:28.865 WARNING:rgw_multi.tests:bucket cngrzz-17 zone test-zone3 behind zone test-zone2: shard=5 master=00000000006.27.9 target=00000000002.19.10 2020-07-13T06:17:33.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:33.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:33.868 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:33.912 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:33.912 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:34.148 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000002.19.10","timestamp":"2020-07-13T06:17:17.373363Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:17:34.149 WARNING:rgw_multi.tests:bucket cngrzz-17 zone test-zone3 behind zone test-zone2: shard=5 master=00000000006.27.9 target=00000000002.19.10 2020-07-13T06:17:39.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:39.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:39.151 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:39.196 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:39.197 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:39.426 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000006.27.9","timestamp":"2020-07-13T06:17:17.247942Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:17:39.427 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-17 2020-07-13T06:17:39.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:39.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:39.484 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:39.528 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:39.730 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.23.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:17:39.730 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-18 2020-07-13T06:17:39.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:39.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:39.732 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:39.816 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:39.816 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:40.020 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"DSATyaKeGjJe.RNQmcXNi5IoXIpCjdP","ns":""},"count":1},"inc_marker":{"position":"00000000002.23.10","timestamp":"2020-07-13T06:17:39.657267Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:17:40.203 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-18 2020-07-13T06:17:40.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:40.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:40.205 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:40.246 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:40.246 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:40.460 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.29.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:17:40.460 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-18 2020-07-13T06:17:40.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:40.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:40.462 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:40.508 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:40.508 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:40.738 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"DSATyaKeGjJe.RNQmcXNi5IoXIpCjdP","ns":""},"count":1},"inc_marker":{"position":"00000000002.23.10","timestamp":"2020-07-13T06:17:39.662160Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:17:40.739 WARNING:rgw_multi.tests:bucket cngrzz-18 zone test-zone2 behind zone test-zone3: shard=5 master=00000000005.29.9 target=00000000002.23.10 2020-07-13T06:17:45.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:45.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:45.742 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:45.791 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:45.791 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:46.031 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"DSATyaKeGjJe.RNQmcXNi5IoXIpCjdP","ns":""},"count":1},"inc_marker":{"position":"00000000002.23.10","timestamp":"2020-07-13T06:17:39.662160Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:17:46.032 WARNING:rgw_multi.tests:bucket cngrzz-18 zone test-zone2 behind zone test-zone3: shard=5 master=00000000005.29.9 target=00000000002.23.10 2020-07-13T06:17:51.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:51.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:51.035 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:51.081 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:51.081 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:51.316 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"DSATyaKeGjJe.RNQmcXNi5IoXIpCjdP","ns":""},"count":1},"inc_marker":{"position":"00000000002.23.10","timestamp":"2020-07-13T06:17:39.662160Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:17:51.317 WARNING:rgw_multi.tests:bucket cngrzz-18 zone test-zone2 behind zone test-zone3: shard=5 master=00000000005.29.9 target=00000000002.23.10 2020-07-13T06:17:56.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:56.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:17:56.321 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:17:56.374 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:17:56.374 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:17:56.606 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"DSATyaKeGjJe.RNQmcXNi5IoXIpCjdP","ns":""},"count":1},"inc_marker":{"position":"00000000005.29.9","timestamp":"2020-07-13T06:17:39.439998Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:17:56.607 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-18 2020-07-13T06:18:00.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:00.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:00.671 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:00.707 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:00.707 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:00.905 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.30.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.12.9"},{"key":4,"val":""},{"key":5,"val":"00000000009.29.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:00.906 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-16 2020-07-13T06:18:00.907 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:00.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:00.908 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:00.953 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:00.953 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:01.197 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y5Ax5e21lW.-n0Vd6cRF.nJjYHIv5JJ","ns":""},"count":1},"inc_marker":{"position":"00000000010.30.5","timestamp":"2020-07-13T06:17:56.739441Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"W-awgo5llJeQnLvWJzL2eCBOVydMd07","ns":""},"count":1},"inc_marker":{"position":"00000000003.12.9","timestamp":"2020-07-13T06:17:56.887981Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000009.29.9","timestamp":"2020-07-13T06:17:58.039688Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:01.198 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-16 2020-07-13T06:18:01.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:01.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:01.200 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:01.247 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:01.248 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:01.477 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.30.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.12.9"},{"key":4,"val":""},{"key":5,"val":"00000000009.29.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:01.477 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-16 2020-07-13T06:18:01.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:01.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:01.479 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:01.567 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:01.567 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:01.808 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y5Ax5e21lW.-n0Vd6cRF.nJjYHIv5JJ","ns":""},"count":1},"inc_marker":{"position":"00000000010.30.5","timestamp":"2020-07-13T06:17:56.739441Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"W-awgo5llJeQnLvWJzL2eCBOVydMd07","ns":""},"count":1},"inc_marker":{"position":"00000000003.12.9","timestamp":"2020-07-13T06:17:56.887981Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000009.29.9","timestamp":"2020-07-13T06:17:58.039688Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:01.808 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-16 2020-07-13T06:18:01.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:01.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:01.810 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:01.856 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:01.857 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:02.077 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.30.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.12.9"},{"key":4,"val":""},{"key":5,"val":"00000000009.29.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:02.077 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-16 2020-07-13T06:18:02.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:02.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:02.079 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:02.126 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:02.126 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:02.401 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.30.5","timestamp":"2020-07-13T06:17:56.739441Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.12.9","timestamp":"2020-07-13T06:17:56.887981Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000009.29.9","timestamp":"2020-07-13T06:17:58.039688Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:02.401 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-16 2020-07-13T06:18:02.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:02.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:02.404 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:02.447 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:02.447 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:02.662 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.19.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.30.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.28.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:02.663 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-16 2020-07-13T06:18:02.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:02.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:02.665 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:02.724 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:02.724 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:02.969 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y5Ax5e21lW.-n0Vd6cRF.nJjYHIv5JJ","ns":""},"count":1},"inc_marker":{"position":"00000000002.17.10","timestamp":"2020-07-13T06:17:56.993447Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"W-awgo5llJeQnLvWJzL2eCBOVydMd07","ns":""},"count":1},"inc_marker":{"position":"00000000010.30.5","timestamp":"2020-07-13T06:17:57.003916Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000006.26.10","timestamp":"2020-07-13T06:17:58.039688Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:02.970 WARNING:rgw_multi.tests:bucket cngrzz-16 zone test-zone1 behind zone test-zone2: shard=1 master=00000000003.19.9 target=00000000002.17.10, shard=5 master=00000000007.28.9 target=00000000006.26.10 2020-07-13T06:18:07.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:07.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:07.974 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:08.020 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:08.021 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:08.268 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y5Ax5e21lW.-n0Vd6cRF.nJjYHIv5JJ","ns":""},"count":1},"inc_marker":{"position":"00000000002.17.10","timestamp":"2020-07-13T06:17:56.993447Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"W-awgo5llJeQnLvWJzL2eCBOVydMd07","ns":""},"count":1},"inc_marker":{"position":"00000000010.30.5","timestamp":"2020-07-13T06:17:57.003916Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000006.26.10","timestamp":"2020-07-13T06:17:58.039688Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:08.269 WARNING:rgw_multi.tests:bucket cngrzz-16 zone test-zone1 behind zone test-zone2: shard=1 master=00000000003.19.9 target=00000000002.17.10, shard=5 master=00000000007.28.9 target=00000000006.26.10 2020-07-13T06:18:13.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:13.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:13.273 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:13.322 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:13.323 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:13.564 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y5Ax5e21lW.-n0Vd6cRF.nJjYHIv5JJ","ns":""},"count":1},"inc_marker":{"position":"00000000002.17.10","timestamp":"2020-07-13T06:17:56.993447Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"W-awgo5llJeQnLvWJzL2eCBOVydMd07","ns":""},"count":1},"inc_marker":{"position":"00000000010.30.5","timestamp":"2020-07-13T06:17:57.003916Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000006.26.10","timestamp":"2020-07-13T06:17:58.039688Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:13.565 WARNING:rgw_multi.tests:bucket cngrzz-16 zone test-zone1 behind zone test-zone2: shard=1 master=00000000003.19.9 target=00000000002.17.10, shard=5 master=00000000007.28.9 target=00000000006.26.10 2020-07-13T06:18:18.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:18.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:18.567 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:18.616 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:18.616 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:18.853 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y5Ax5e21lW.-n0Vd6cRF.nJjYHIv5JJ","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-07-13T06:17:56.616494Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"W-awgo5llJeQnLvWJzL2eCBOVydMd07","ns":""},"count":1},"inc_marker":{"position":"00000000010.30.5","timestamp":"2020-07-13T06:17:57.003916Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000007.28.9","timestamp":"2020-07-13T06:17:58.039688Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:18.854 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-16 2020-07-13T06:18:18.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:18.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:18.856 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:18.904 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:18.905 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:19.133 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.19.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.30.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.28.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:19.134 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-16 2020-07-13T06:18:19.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:19.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:19.136 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:19.181 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:19.181 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:19.432 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y5Ax5e21lW.-n0Vd6cRF.nJjYHIv5JJ","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-07-13T06:17:56.616494Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"W-awgo5llJeQnLvWJzL2eCBOVydMd07","ns":""},"count":1},"inc_marker":{"position":"00000000010.30.5","timestamp":"2020-07-13T06:17:57.003916Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000007.28.9","timestamp":"2020-07-13T06:17:58.039688Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:19.434 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-16 2020-07-13T06:18:19.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:19.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:19.436 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:19.479 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:19.479 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:19.682 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.17.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.15.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.43.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:19.683 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-16 2020-07-13T06:18:19.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:19.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:19.685 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:19.770 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:19.771 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:20.026 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y5Ax5e21lW.-n0Vd6cRF.nJjYHIv5JJ","ns":""},"count":1},"inc_marker":{"position":"00000000003.17.9","timestamp":"2020-07-13T06:17:56.616494Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"W-awgo5llJeQnLvWJzL2eCBOVydMd07","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9","timestamp":"2020-07-13T06:17:56.887981Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000014.43.5","timestamp":"2020-07-13T06:17:58.167951Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:20.026 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-16 2020-07-13T06:18:20.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:20.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:20.028 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:20.070 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:20.071 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:20.287 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.17.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.15.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.43.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:20.288 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-16 2020-07-13T06:18:20.289 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:20.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:20.290 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:20.334 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:20.335 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:20.574 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y5Ax5e21lW.-n0Vd6cRF.nJjYHIv5JJ","ns":""},"count":1},"inc_marker":{"position":"00000000003.17.9","timestamp":"2020-07-13T06:17:56.616494Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"W-awgo5llJeQnLvWJzL2eCBOVydMd07","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9","timestamp":"2020-07-13T06:17:56.887981Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000014.43.5","timestamp":"2020-07-13T06:17:58.167951Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:20.575 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-16 2020-07-13T06:18:20.576 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-16 zones={test-zone2, test-zone1} 2020-07-13T06:18:21.462 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:18:21.471 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:18:21.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:18:21.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:18:21.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:18:21.474 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:18:21.482 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:18:21.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:18:21.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:18:21.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:18:21.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:21.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:21.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:21.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:21.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:21.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:21.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:22.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:22.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:22.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:22.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:22.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:22.528 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-16 zones={test-zone2, test-zone1} 2020-07-13T06:18:22.528 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-16 zones={test-zone3, test-zone1} 2020-07-13T06:18:23.498 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:18:23.505 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:18:23.506 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:18:23.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:18:23.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:18:23.508 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:18:23.524 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:18:23.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:18:23.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:18:23.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:18:23.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:23.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:23.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:23.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:23.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:23.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:23.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:24.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:24.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:24.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:24.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:24.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:24.555 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-16 zones={test-zone3, test-zone1} 2020-07-13T06:18:24.556 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-16 zones={test-zone3, test-zone2} 2020-07-13T06:18:24.585 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:18:24.594 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:18:24.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:18:24.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:18:24.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:18:24.597 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:18:24.606 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:18:24.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:18:24.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:18:24.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:18:24.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:24.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:24.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:24.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:24.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:24.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:25.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:25.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:25.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:25.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:25.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:25.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:25.613 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-16 zones={test-zone3, test-zone2} 2020-07-13T06:18:25.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:25.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:25.615 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:25.657 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:25.657 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:25.890 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.29.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.14.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.29.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:25.890 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-17 2020-07-13T06:18:25.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:25.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:25.892 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:25.938 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:25.938 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:26.172 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"zLXY27CAzz2JEah8dgxAenqUW3J8GO1","ns":""},"count":1},"inc_marker":{"position":"00000000010.29.5","timestamp":"2020-07-13T06:17:59.314168Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"u0.xQPl7wQXSL5SsrbgaLFb2lUt6UB6","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9","timestamp":"2020-07-13T06:17:59.496747Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.9","timestamp":"2020-07-13T06:17:17.247942Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:26.173 WARNING:rgw_multi.tests:bucket cngrzz-17 zone test-zone2 behind zone test-zone1: shard=5 master=00000000007.29.9 target=00000000004.23.9 2020-07-13T06:18:31.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:31.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:31.177 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:31.223 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:31.223 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:31.458 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"zLXY27CAzz2JEah8dgxAenqUW3J8GO1","ns":""},"count":1},"inc_marker":{"position":"00000000010.29.5","timestamp":"2020-07-13T06:17:59.314168Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"u0.xQPl7wQXSL5SsrbgaLFb2lUt6UB6","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9","timestamp":"2020-07-13T06:17:59.496747Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.9","timestamp":"2020-07-13T06:17:17.247942Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:31.458 WARNING:rgw_multi.tests:bucket cngrzz-17 zone test-zone2 behind zone test-zone1: shard=5 master=00000000007.29.9 target=00000000004.23.9 2020-07-13T06:18:36.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:36.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:36.463 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:36.509 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:36.509 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:36.734 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"zLXY27CAzz2JEah8dgxAenqUW3J8GO1","ns":""},"count":1},"inc_marker":{"position":"00000000010.29.5","timestamp":"2020-07-13T06:17:59.314168Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"u0.xQPl7wQXSL5SsrbgaLFb2lUt6UB6","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9","timestamp":"2020-07-13T06:17:59.496747Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000007.29.9","timestamp":"2020-07-13T06:17:59.878202Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:36.735 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-17 2020-07-13T06:18:36.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:36.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:36.737 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:36.778 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:36.779 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:37.011 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.29.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.14.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.29.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:37.011 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-17 2020-07-13T06:18:37.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:37.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:37.013 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:37.095 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:37.095 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:37.333 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"zLXY27CAzz2JEah8dgxAenqUW3J8GO1","ns":""},"count":1},"inc_marker":{"position":"00000000010.29.5","timestamp":"2020-07-13T06:17:59.313310Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"u0.xQPl7wQXSL5SsrbgaLFb2lUt6UB6","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9","timestamp":"2020-07-13T06:17:59.496747Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000007.29.9","timestamp":"2020-07-13T06:17:59.878202Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:37.333 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-17 2020-07-13T06:18:37.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:37.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:37.335 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:37.418 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:37.418 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:37.630 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.29.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.14.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.29.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:37.631 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-17 2020-07-13T06:18:37.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:37.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:37.633 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:37.683 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:37.683 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:37.954 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.29.5","timestamp":"2020-07-13T06:17:59.194694Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.14.9","timestamp":"2020-07-13T06:17:59.496747Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.29.9","timestamp":"2020-07-13T06:17:59.878202Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:37.955 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-17 2020-07-13T06:18:37.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:37.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:37.956 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:38.036 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:38.037 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:38.260 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.13.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.32.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.33.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:38.261 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-17 2020-07-13T06:18:38.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:38.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:38.263 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:38.309 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:38.310 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:38.561 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"zLXY27CAzz2JEah8dgxAenqUW3J8GO1","ns":""},"count":1},"inc_marker":{"position":"00000000003.13.9","timestamp":"2020-07-13T06:17:59.073240Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"u0.xQPl7wQXSL5SsrbgaLFb2lUt6UB6","ns":""},"count":1},"inc_marker":{"position":"00000000010.32.5","timestamp":"2020-07-13T06:17:59.611918Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000009.33.9","timestamp":"2020-07-13T06:17:59.878202Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:38.561 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-17 2020-07-13T06:18:38.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:38.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:38.563 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:38.614 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:38.615 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:38.827 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.13.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.32.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.33.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:38.828 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-17 2020-07-13T06:18:38.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:38.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:38.830 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:38.874 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:38.875 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:39.112 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"zLXY27CAzz2JEah8dgxAenqUW3J8GO1","ns":""},"count":1},"inc_marker":{"position":"00000000003.13.9","timestamp":"2020-07-13T06:17:59.073240Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"u0.xQPl7wQXSL5SsrbgaLFb2lUt6UB6","ns":""},"count":1},"inc_marker":{"position":"00000000010.32.5","timestamp":"2020-07-13T06:17:59.611918Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000009.33.9","timestamp":"2020-07-13T06:17:59.878202Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:39.143 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-17 2020-07-13T06:18:39.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:39.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:39.145 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:39.198 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:39.199 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:39.432 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.15.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.15.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.41.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:39.432 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-17 2020-07-13T06:18:39.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:39.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:39.434 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:39.524 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:39.525 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:39.738 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"zLXY27CAzz2JEah8dgxAenqUW3J8GO1","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9","timestamp":"2020-07-13T06:17:59.073240Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"u0.xQPl7wQXSL5SsrbgaLFb2lUt6UB6","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9","timestamp":"2020-07-13T06:17:59.496747Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000014.41.5","timestamp":"2020-07-13T06:18:00.000973Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:39.739 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-17 2020-07-13T06:18:39.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:39.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:39.741 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:39.782 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:39.783 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:39.999 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.15.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.15.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.41.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:40.000 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-17 2020-07-13T06:18:40.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:40.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:40.002 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:40.046 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:40.047 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:40.278 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"zLXY27CAzz2JEah8dgxAenqUW3J8GO1","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9","timestamp":"2020-07-13T06:17:59.073240Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"u0.xQPl7wQXSL5SsrbgaLFb2lUt6UB6","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9","timestamp":"2020-07-13T06:17:59.496747Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000014.41.5","timestamp":"2020-07-13T06:18:00.000973Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:40.279 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-17 2020-07-13T06:18:40.280 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-17 zones={test-zone2, test-zone1} 2020-07-13T06:18:41.725 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:18:41.742 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:18:41.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:18:41.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:18:41.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:18:41.745 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:18:41.755 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:18:41.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:18:41.756 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:18:41.756 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:18:41.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:41.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:41.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:41.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:41.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:42.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:42.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:42.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:42.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:42.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:42.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:42.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:42.808 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-17 zones={test-zone2, test-zone1} 2020-07-13T06:18:42.809 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-17 zones={test-zone3, test-zone1} 2020-07-13T06:18:42.840 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:18:42.848 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:18:42.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:18:42.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:18:42.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:18:42.850 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:18:42.857 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:18:42.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:18:42.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:18:42.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:18:42.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:42.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:42.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:43.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:43.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:43.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:43.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:43.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:43.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:43.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:43.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:43.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:43.939 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-17 zones={test-zone3, test-zone1} 2020-07-13T06:18:43.939 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-17 zones={test-zone3, test-zone2} 2020-07-13T06:18:44.699 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:18:44.709 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:18:44.710 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:18:44.710 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:18:44.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:18:44.711 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:18:44.731 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:18:44.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:18:44.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:18:44.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:18:44.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:44.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:44.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:44.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:44.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:45.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:45.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:45.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:45.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:45.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:45.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:45.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:45.754 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-17 zones={test-zone3, test-zone2} 2020-07-13T06:18:45.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:45.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:45.756 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:45.803 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:45.804 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:46.038 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.23.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.18.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.30.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:46.039 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-18 2020-07-13T06:18:46.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:46.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:46.041 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:46.086 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:46.086 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:46.325 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sWHb7Y70Sihw1xLvS6ZOH67P6q20MQs","ns":""},"count":1},"inc_marker":{"position":"00000000010.23.5","timestamp":"2020-07-13T06:18:00.269390Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4v6d6IlBSfq7i.DeG0IWqYgqQDBZTSi","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.9","timestamp":"2020-07-13T06:18:00.279788Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"DSATyaKeGjJe.RNQmcXNi5IoXIpCjdP","ns":""},"count":1},"inc_marker":{"position":"00000000007.30.9","timestamp":"2020-07-13T06:18:00.472724Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:46.326 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-18 2020-07-13T06:18:46.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:46.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:46.327 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:46.369 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:46.369 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:46.603 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.23.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.18.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.30.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:46.604 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-18 2020-07-13T06:18:46.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:46.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:46.606 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:46.688 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:46.688 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:46.935 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sWHb7Y70Sihw1xLvS6ZOH67P6q20MQs","ns":""},"count":1},"inc_marker":{"position":"00000000010.23.5","timestamp":"2020-07-13T06:18:00.197483Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4v6d6IlBSfq7i.DeG0IWqYgqQDBZTSi","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.9","timestamp":"2020-07-13T06:18:00.279788Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"DSATyaKeGjJe.RNQmcXNi5IoXIpCjdP","ns":""},"count":1},"inc_marker":{"position":"00000000007.30.9","timestamp":"2020-07-13T06:18:00.472724Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:46.935 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-18 2020-07-13T06:18:46.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:46.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:46.937 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:47.016 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:47.016 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:47.249 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.23.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.18.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.30.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:47.256 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-18 2020-07-13T06:18:47.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:47.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:47.258 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:47.311 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:47.312 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:47.590 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.23.5","timestamp":"2020-07-13T06:18:00.197483Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.18.9","timestamp":"2020-07-13T06:18:00.279788Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.30.9","timestamp":"2020-07-13T06:18:00.472724Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:47.590 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-18 2020-07-13T06:18:47.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:47.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:47.592 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:47.700 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:47.700 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:47.927 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.14.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.53.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.29.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:47.928 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-18 2020-07-13T06:18:47.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:47.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:47.930 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:47.976 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:47.976 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:48.232 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sWHb7Y70Sihw1xLvS6ZOH67P6q20MQs","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9","timestamp":"2020-07-13T06:18:00.074099Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4v6d6IlBSfq7i.DeG0IWqYgqQDBZTSi","ns":""},"count":1},"inc_marker":{"position":"00000000010.53.5","timestamp":"2020-07-13T06:18:00.392970Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"DSATyaKeGjJe.RNQmcXNi5IoXIpCjdP","ns":""},"count":1},"inc_marker":{"position":"00000000007.29.9","timestamp":"2020-07-13T06:18:00.472724Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:48.233 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-18 2020-07-13T06:18:48.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:48.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:48.234 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:48.279 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:48.281 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:48.519 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.14.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.53.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.29.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:48.520 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-18 2020-07-13T06:18:48.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:48.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:48.523 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:48.575 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:48.575 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:48.817 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sWHb7Y70Sihw1xLvS6ZOH67P6q20MQs","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9","timestamp":"2020-07-13T06:18:00.074099Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4v6d6IlBSfq7i.DeG0IWqYgqQDBZTSi","ns":""},"count":1},"inc_marker":{"position":"00000000010.53.5","timestamp":"2020-07-13T06:18:00.392970Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"DSATyaKeGjJe.RNQmcXNi5IoXIpCjdP","ns":""},"count":1},"inc_marker":{"position":"00000000007.29.9","timestamp":"2020-07-13T06:18:00.472724Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:48.818 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-18 2020-07-13T06:18:48.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:48.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:48.819 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:48.862 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:48.862 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:49.099 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.15.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.21.9"},{"key":4,"val":""},{"key":5,"val":"00000000016.51.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:49.100 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-18 2020-07-13T06:18:49.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:49.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:49.102 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:49.185 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:49.185 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:49.452 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sWHb7Y70Sihw1xLvS6ZOH67P6q20MQs","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9","timestamp":"2020-07-13T06:18:00.074099Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4v6d6IlBSfq7i.DeG0IWqYgqQDBZTSi","ns":""},"count":1},"inc_marker":{"position":"00000000003.21.9","timestamp":"2020-07-13T06:18:00.279788Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"DSATyaKeGjJe.RNQmcXNi5IoXIpCjdP","ns":""},"count":1},"inc_marker":{"position":"00000000016.51.5","timestamp":"2020-07-13T06:18:00.592922Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:49.452 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-18 2020-07-13T06:18:49.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:49.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:49.454 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:49.532 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:49.533 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:49.770 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.15.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.21.9"},{"key":4,"val":""},{"key":5,"val":"00000000016.51.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:49.771 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-18 2020-07-13T06:18:49.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:49.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:49.773 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:49.819 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:49.819 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:50.061 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sWHb7Y70Sihw1xLvS6ZOH67P6q20MQs","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9","timestamp":"2020-07-13T06:18:00.074099Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4v6d6IlBSfq7i.DeG0IWqYgqQDBZTSi","ns":""},"count":1},"inc_marker":{"position":"00000000003.21.9","timestamp":"2020-07-13T06:18:00.279788Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"DSATyaKeGjJe.RNQmcXNi5IoXIpCjdP","ns":""},"count":1},"inc_marker":{"position":"00000000016.51.5","timestamp":"2020-07-13T06:18:00.592922Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:50.061 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-18 2020-07-13T06:18:50.062 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-18 zones={test-zone2, test-zone1} 2020-07-13T06:18:50.852 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:18:50.862 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:18:50.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:18:50.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:18:50.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:18:50.866 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:18:50.876 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:18:50.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:18:50.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:18:50.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:18:50.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:50.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:50.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:51.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:51.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:51.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:51.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:51.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:51.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:51.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:51.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:51.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:51.903 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-18 zones={test-zone2, test-zone1} 2020-07-13T06:18:51.904 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-18 zones={test-zone3, test-zone1} 2020-07-13T06:18:52.124 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:18:52.131 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:18:52.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:18:52.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:18:52.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:18:52.133 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:18:52.140 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:18:52.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:18:52.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:18:52.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:18:52.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:52.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:52.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:52.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:52.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:52.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:52.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:52.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:52.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:52.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:52.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:53.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:53.150 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-18 zones={test-zone3, test-zone1} 2020-07-13T06:18:53.151 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-18 zones={test-zone3, test-zone2} 2020-07-13T06:18:53.911 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:18:53.923 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:18:53.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:18:53.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:18:53.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:18:53.925 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:18:53.932 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:18:53.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:18:53.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:18:53.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:18:53.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:53.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:54.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:18:54.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:54.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:54.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:18:54.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:54.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:54.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:18:54.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:54.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:54.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:18:54.950 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-18 zones={test-zone3, test-zone2} 2020-07-13T06:18:55.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:55.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:55.684 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:55.727 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:55.727 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:55.931 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.37.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.12.9"},{"key":4,"val":""},{"key":5,"val":"00000000010.32.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:18:55.931 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-16 2020-07-13T06:18:55.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:55.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:18:55.933 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:18:55.979 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:18:55.980 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:18:56.213 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y5Ax5e21lW.-n0Vd6cRF.nJjYHIv5JJ","ns":""},"count":1},"inc_marker":{"position":"00000000010.30.5","timestamp":"2020-07-13T06:17:56.739441Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"W-awgo5llJeQnLvWJzL2eCBOVydMd07","ns":""},"count":1},"inc_marker":{"position":"00000000003.12.9","timestamp":"2020-07-13T06:17:56.887981Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000010.32.5","timestamp":"2020-07-13T06:18:55.233349Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:18:56.214 WARNING:rgw_multi.tests:bucket cngrzz-16 zone test-zone2 behind zone test-zone1: shard=1 master=00000000011.37.5 target=00000000010.30.5 2020-07-13T06:19:01.215 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:01.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:01.217 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:01.264 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:01.265 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:01.503 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y5Ax5e21lW.-n0Vd6cRF.nJjYHIv5JJ","ns":""},"count":1},"inc_marker":{"position":"00000000011.37.5","timestamp":"2020-07-13T06:18:55.089225Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"W-awgo5llJeQnLvWJzL2eCBOVydMd07","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.5","timestamp":"2020-07-13T06:18:55.160281Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000010.32.5","timestamp":"2020-07-13T06:18:55.233349Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:01.504 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-16 2020-07-13T06:19:01.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:01.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:01.506 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:01.552 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:01.552 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:01.775 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.37.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.21.5"},{"key":4,"val":""},{"key":5,"val":"00000000010.32.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:01.776 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-16 2020-07-13T06:19:01.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:01.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:01.779 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:01.862 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:01.862 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:02.102 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y5Ax5e21lW.-n0Vd6cRF.nJjYHIv5JJ","ns":""},"count":1},"inc_marker":{"position":"00000000011.37.5","timestamp":"2020-07-13T06:18:55.089225Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"W-awgo5llJeQnLvWJzL2eCBOVydMd07","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.5","timestamp":"2020-07-13T06:18:55.160281Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000010.32.5","timestamp":"2020-07-13T06:18:55.233349Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:02.103 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-16 2020-07-13T06:19:02.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:02.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:02.105 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:02.147 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:02.147 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:02.380 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.37.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.21.5"},{"key":4,"val":""},{"key":5,"val":"00000000010.32.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:02.381 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-16 2020-07-13T06:19:02.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:02.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:02.383 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:02.428 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:02.428 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:02.702 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.37.5","timestamp":"2020-07-13T06:18:55.089225Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.21.5","timestamp":"2020-07-13T06:18:55.160281Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.32.5","timestamp":"2020-07-13T06:18:55.233349Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:02.703 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-16 2020-07-13T06:19:02.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:02.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:02.705 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:02.787 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:02.787 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:03.015 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.22.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.61.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.31.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:03.015 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-16 2020-07-13T06:19:03.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:03.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:03.017 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:03.062 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:03.063 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:03.308 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y5Ax5e21lW.-n0Vd6cRF.nJjYHIv5JJ","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.5","timestamp":"2020-07-13T06:18:55.089225Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"W-awgo5llJeQnLvWJzL2eCBOVydMd07","ns":""},"count":1},"inc_marker":{"position":"00000000011.61.5","timestamp":"2020-07-13T06:18:55.160281Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000008.31.5","timestamp":"2020-07-13T06:18:55.233349Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:03.309 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-16 2020-07-13T06:19:03.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:03.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:03.310 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:03.354 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:03.355 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:03.595 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.22.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.61.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.31.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:03.595 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-16 2020-07-13T06:19:03.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:03.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:03.597 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:03.643 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:03.644 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:03.895 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y5Ax5e21lW.-n0Vd6cRF.nJjYHIv5JJ","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.5","timestamp":"2020-07-13T06:18:55.089225Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"W-awgo5llJeQnLvWJzL2eCBOVydMd07","ns":""},"count":1},"inc_marker":{"position":"00000000011.61.5","timestamp":"2020-07-13T06:18:55.160281Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000008.31.5","timestamp":"2020-07-13T06:18:55.233349Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:03.895 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-16 2020-07-13T06:19:03.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:03.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:03.898 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:03.939 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:03.940 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:04.176 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.20.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.26.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.46.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:04.176 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-16 2020-07-13T06:19:04.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:04.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:04.178 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:04.262 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:04.262 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:04.503 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y5Ax5e21lW.-n0Vd6cRF.nJjYHIv5JJ","ns":""},"count":1},"inc_marker":{"position":"00000000004.20.5","timestamp":"2020-07-13T06:18:55.089225Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"W-awgo5llJeQnLvWJzL2eCBOVydMd07","ns":""},"count":1},"inc_marker":{"position":"00000000004.26.5","timestamp":"2020-07-13T06:18:55.160281Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000015.46.5","timestamp":"2020-07-13T06:18:55.233349Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:04.503 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-16 2020-07-13T06:19:04.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:04.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:04.505 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:04.593 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:04.593 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:04.808 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.20.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.26.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.46.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:04.808 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-16 2020-07-13T06:19:04.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:04.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:04.810 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:04.857 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:04.858 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:05.110 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y5Ax5e21lW.-n0Vd6cRF.nJjYHIv5JJ","ns":""},"count":1},"inc_marker":{"position":"00000000004.20.5","timestamp":"2020-07-13T06:18:55.089225Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"W-awgo5llJeQnLvWJzL2eCBOVydMd07","ns":""},"count":1},"inc_marker":{"position":"00000000004.26.5","timestamp":"2020-07-13T06:18:55.160281Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"Wxcqgp4wLc9dn.VpE9bUZcHfgvrxLOC","ns":""},"count":1},"inc_marker":{"position":"00000000015.46.5","timestamp":"2020-07-13T06:18:55.233349Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:05.111 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-16 2020-07-13T06:19:05.111 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-16 zones={test-zone2, test-zone1} 2020-07-13T06:19:06.014 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:19:06.022 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:19:06.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:19:06.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:19:06.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:19:06.024 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:19:06.035 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:19:06.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:19:06.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:19:06.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:19:06.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:06.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:06.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:06.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:06.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:06.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:06.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:06.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:06.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:06.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:06.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:06.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:07.047 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-16 zones={test-zone2, test-zone1} 2020-07-13T06:19:07.048 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-16 zones={test-zone3, test-zone1} 2020-07-13T06:19:07.932 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:19:07.938 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:19:07.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:19:07.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:19:07.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:19:07.940 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:19:07.947 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:19:07.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:19:07.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:19:07.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:19:07.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:07.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:08.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:08.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:08.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:08.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:08.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:08.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:08.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:08.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:08.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:08.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:08.930 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-16 zones={test-zone3, test-zone1} 2020-07-13T06:19:08.931 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-16 zones={test-zone3, test-zone2} 2020-07-13T06:19:09.363 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:19:09.374 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:19:09.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:19:09.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:19:09.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:19:09.376 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:19:09.382 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:19:09.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:19:09.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:19:09.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:19:09.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:09.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:09.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:09.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:09.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:09.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:09.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:09.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:09.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:10.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:10.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:10.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:10.406 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-16 zones={test-zone3, test-zone2} 2020-07-13T06:19:10.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:10.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:10.409 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:10.451 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:10.452 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:10.683 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.36.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.17.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.32.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:10.684 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-17 2020-07-13T06:19:10.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:10.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:10.687 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:10.733 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:10.733 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:10.979 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"zLXY27CAzz2JEah8dgxAenqUW3J8GO1","ns":""},"count":1},"inc_marker":{"position":"00000000011.36.5","timestamp":"2020-07-13T06:18:55.319326Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"u0.xQPl7wQXSL5SsrbgaLFb2lUt6UB6","ns":""},"count":1},"inc_marker":{"position":"00000000004.17.5","timestamp":"2020-07-13T06:18:55.389775Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000008.32.5","timestamp":"2020-07-13T06:18:55.458467Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:10.984 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-17 2020-07-13T06:19:10.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:10.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:10.987 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:11.032 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:11.033 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:11.260 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.36.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.17.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.32.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:11.260 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-17 2020-07-13T06:19:11.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:11.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:11.262 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:11.345 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:11.345 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:11.580 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"zLXY27CAzz2JEah8dgxAenqUW3J8GO1","ns":""},"count":1},"inc_marker":{"position":"00000000011.36.5","timestamp":"2020-07-13T06:18:55.319326Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"u0.xQPl7wQXSL5SsrbgaLFb2lUt6UB6","ns":""},"count":1},"inc_marker":{"position":"00000000004.17.5","timestamp":"2020-07-13T06:18:55.389775Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000008.32.5","timestamp":"2020-07-13T06:18:55.458467Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:11.581 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-17 2020-07-13T06:19:11.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:11.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:11.584 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:11.660 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:11.661 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:11.882 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.36.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.17.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.32.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:11.883 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-17 2020-07-13T06:19:11.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:11.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:11.885 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:11.928 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:11.929 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:12.200 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.36.5","timestamp":"2020-07-13T06:18:55.319326Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.17.5","timestamp":"2020-07-13T06:18:55.389775Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.32.5","timestamp":"2020-07-13T06:18:55.458467Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:12.200 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-17 2020-07-13T06:19:12.201 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:12.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:12.202 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:12.283 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:12.283 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:12.507 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.16.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.39.5"},{"key":4,"val":""},{"key":5,"val":"00000000010.36.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:12.507 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-17 2020-07-13T06:19:12.508 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:12.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:12.509 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:12.556 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:12.556 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:12.795 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"zLXY27CAzz2JEah8dgxAenqUW3J8GO1","ns":""},"count":1},"inc_marker":{"position":"00000000004.16.5","timestamp":"2020-07-13T06:18:55.319326Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"u0.xQPl7wQXSL5SsrbgaLFb2lUt6UB6","ns":""},"count":1},"inc_marker":{"position":"00000000011.39.5","timestamp":"2020-07-13T06:18:55.389775Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000010.36.5","timestamp":"2020-07-13T06:18:55.458467Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:12.796 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-17 2020-07-13T06:19:12.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:12.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:12.798 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:12.840 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:12.840 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:13.063 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.16.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.39.5"},{"key":4,"val":""},{"key":5,"val":"00000000010.36.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:13.064 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-17 2020-07-13T06:19:13.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:13.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:13.069 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:13.134 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:13.135 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:13.386 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"zLXY27CAzz2JEah8dgxAenqUW3J8GO1","ns":""},"count":1},"inc_marker":{"position":"00000000004.16.5","timestamp":"2020-07-13T06:18:55.319326Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"u0.xQPl7wQXSL5SsrbgaLFb2lUt6UB6","ns":""},"count":1},"inc_marker":{"position":"00000000011.39.5","timestamp":"2020-07-13T06:18:55.389775Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000010.36.5","timestamp":"2020-07-13T06:18:55.458467Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:13.386 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-17 2020-07-13T06:19:13.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:13.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:13.388 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:13.470 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:13.471 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:13.683 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.18.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.18.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.44.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:13.684 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-17 2020-07-13T06:19:13.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:13.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:13.685 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:13.770 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:13.770 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:14.003 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"zLXY27CAzz2JEah8dgxAenqUW3J8GO1","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5","timestamp":"2020-07-13T06:18:55.319326Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"u0.xQPl7wQXSL5SsrbgaLFb2lUt6UB6","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5","timestamp":"2020-07-13T06:18:55.389775Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000015.44.5","timestamp":"2020-07-13T06:18:55.458467Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:14.004 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-17 2020-07-13T06:19:14.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:14.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:14.006 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:14.046 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:14.046 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:14.293 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.18.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.18.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.44.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:14.294 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-17 2020-07-13T06:19:14.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:14.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:14.296 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:14.340 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:14.340 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:14.589 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"zLXY27CAzz2JEah8dgxAenqUW3J8GO1","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5","timestamp":"2020-07-13T06:18:55.319326Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"u0.xQPl7wQXSL5SsrbgaLFb2lUt6UB6","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5","timestamp":"2020-07-13T06:18:55.389775Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E8-SBeZlP34ADU.YJHSr4yimKsid888","ns":""},"count":1},"inc_marker":{"position":"00000000015.44.5","timestamp":"2020-07-13T06:18:55.458467Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:14.589 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-17 2020-07-13T06:19:14.590 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-17 zones={test-zone2, test-zone1} 2020-07-13T06:19:15.562 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:19:15.571 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:19:15.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:19:15.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:19:15.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:19:15.573 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:19:15.581 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:19:15.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:19:15.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:19:15.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:19:15.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:15.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:15.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:15.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:15.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:15.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:16.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:16.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:16.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:16.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:16.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:16.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:16.613 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-17 zones={test-zone2, test-zone1} 2020-07-13T06:19:16.614 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-17 zones={test-zone3, test-zone1} 2020-07-13T06:19:17.529 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:19:17.536 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:19:17.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:19:17.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:19:17.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:19:17.538 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:19:17.546 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:19:17.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:19:17.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:19:17.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:19:17.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:17.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:17.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:17.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:17.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:17.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:18.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:18.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:18.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:18.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:18.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:18.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:18.587 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-17 zones={test-zone3, test-zone1} 2020-07-13T06:19:18.588 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-17 zones={test-zone3, test-zone2} 2020-07-13T06:19:19.415 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:19:19.426 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:19:19.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:19:19.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:19:19.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:19:19.428 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:19:19.436 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:19:19.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:19:19.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:19:19.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:19:19.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:19.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:19.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:19.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:19.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:19.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:19.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:19.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:20.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:20.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:20.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:20.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:20.454 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-17 zones={test-zone3, test-zone2} 2020-07-13T06:19:20.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:20.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:20.456 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:20.498 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:20.499 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:20.704 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.30.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.23.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.33.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:20.704 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-18 2020-07-13T06:19:20.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:20.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:20.706 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:20.750 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:20.751 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:20.996 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sWHb7Y70Sihw1xLvS6ZOH67P6q20MQs","ns":""},"count":1},"inc_marker":{"position":"00000000011.30.5","timestamp":"2020-07-13T06:18:55.530260Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4v6d6IlBSfq7i.DeG0IWqYgqQDBZTSi","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.5","timestamp":"2020-07-13T06:18:55.598904Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"DSATyaKeGjJe.RNQmcXNi5IoXIpCjdP","ns":""},"count":1},"inc_marker":{"position":"00000000008.33.5","timestamp":"2020-07-13T06:18:55.669364Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:20.997 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-18 2020-07-13T06:19:20.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:20.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:20.999 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:21.040 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:21.040 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:21.263 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.30.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.23.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.33.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:21.264 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-18 2020-07-13T06:19:21.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:21.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:21.265 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:21.351 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:21.351 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:21.596 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sWHb7Y70Sihw1xLvS6ZOH67P6q20MQs","ns":""},"count":1},"inc_marker":{"position":"00000000011.30.5","timestamp":"2020-07-13T06:18:55.530260Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4v6d6IlBSfq7i.DeG0IWqYgqQDBZTSi","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.5","timestamp":"2020-07-13T06:18:55.598904Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"DSATyaKeGjJe.RNQmcXNi5IoXIpCjdP","ns":""},"count":1},"inc_marker":{"position":"00000000008.33.5","timestamp":"2020-07-13T06:18:55.669364Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:21.596 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-18 2020-07-13T06:19:21.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:21.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:21.598 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:21.635 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:21.636 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:21.841 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.30.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.23.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.33.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:21.842 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-18 2020-07-13T06:19:21.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:21.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:21.844 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:21.891 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:21.892 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:22.162 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.30.5","timestamp":"2020-07-13T06:18:55.530260Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.23.5","timestamp":"2020-07-13T06:18:55.598904Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.33.5","timestamp":"2020-07-13T06:18:55.669364Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:22.163 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-18 2020-07-13T06:19:22.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:22.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:22.164 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:22.207 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:22.208 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:22.446 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.17.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.63.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.32.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:22.446 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-18 2020-07-13T06:19:22.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:22.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:22.448 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:22.491 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:22.492 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:22.722 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sWHb7Y70Sihw1xLvS6ZOH67P6q20MQs","ns":""},"count":1},"inc_marker":{"position":"00000000004.17.5","timestamp":"2020-07-13T06:18:55.530260Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4v6d6IlBSfq7i.DeG0IWqYgqQDBZTSi","ns":""},"count":1},"inc_marker":{"position":"00000000011.63.5","timestamp":"2020-07-13T06:18:55.598904Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"DSATyaKeGjJe.RNQmcXNi5IoXIpCjdP","ns":""},"count":1},"inc_marker":{"position":"00000000008.32.5","timestamp":"2020-07-13T06:18:55.669364Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:22.723 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-18 2020-07-13T06:19:22.723 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:22.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:22.725 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:22.767 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:22.767 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:22.989 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.17.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.63.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.32.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:22.990 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-18 2020-07-13T06:19:22.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:22.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:22.992 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:23.035 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:23.035 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:23.267 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sWHb7Y70Sihw1xLvS6ZOH67P6q20MQs","ns":""},"count":1},"inc_marker":{"position":"00000000004.17.5","timestamp":"2020-07-13T06:18:55.530260Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4v6d6IlBSfq7i.DeG0IWqYgqQDBZTSi","ns":""},"count":1},"inc_marker":{"position":"00000000011.63.5","timestamp":"2020-07-13T06:18:55.598904Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"DSATyaKeGjJe.RNQmcXNi5IoXIpCjdP","ns":""},"count":1},"inc_marker":{"position":"00000000008.32.5","timestamp":"2020-07-13T06:18:55.669364Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:23.268 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-18 2020-07-13T06:19:23.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:23.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:23.270 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:23.349 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:23.350 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:23.562 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.18.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.24.5"},{"key":4,"val":""},{"key":5,"val":"00000000017.54.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:23.563 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-18 2020-07-13T06:19:23.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:23.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:23.565 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:23.648 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:23.649 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:23.894 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sWHb7Y70Sihw1xLvS6ZOH67P6q20MQs","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5","timestamp":"2020-07-13T06:18:55.530260Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4v6d6IlBSfq7i.DeG0IWqYgqQDBZTSi","ns":""},"count":1},"inc_marker":{"position":"00000000004.24.5","timestamp":"2020-07-13T06:18:55.598904Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"DSATyaKeGjJe.RNQmcXNi5IoXIpCjdP","ns":""},"count":1},"inc_marker":{"position":"00000000017.54.5","timestamp":"2020-07-13T06:18:55.669364Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:23.895 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-18 2020-07-13T06:19:23.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:23.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:23.896 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:23.979 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:23.979 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:24.197 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.18.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.24.5"},{"key":4,"val":""},{"key":5,"val":"00000000017.54.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:24.197 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-18 2020-07-13T06:19:24.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:24.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:24.199 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:24.250 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:24.251 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:24.500 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sWHb7Y70Sihw1xLvS6ZOH67P6q20MQs","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5","timestamp":"2020-07-13T06:18:55.530260Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4v6d6IlBSfq7i.DeG0IWqYgqQDBZTSi","ns":""},"count":1},"inc_marker":{"position":"00000000004.24.5","timestamp":"2020-07-13T06:18:55.598904Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"DSATyaKeGjJe.RNQmcXNi5IoXIpCjdP","ns":""},"count":1},"inc_marker":{"position":"00000000017.54.5","timestamp":"2020-07-13T06:18:55.669364Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:24.501 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-18 2020-07-13T06:19:24.501 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-18 zones={test-zone2, test-zone1} 2020-07-13T06:19:25.739 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:19:25.748 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:19:25.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:19:25.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:19:25.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:19:25.750 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:19:25.759 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:19:25.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:19:25.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:19:25.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:19:25.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:25.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:25.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:25.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:25.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:26.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:26.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:26.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:26.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:26.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:26.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:26.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:26.805 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-18 zones={test-zone2, test-zone1} 2020-07-13T06:19:26.805 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-18 zones={test-zone3, test-zone1} 2020-07-13T06:19:26.913 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:19:26.920 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:19:26.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:19:26.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:19:26.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:19:26.922 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:19:26.928 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:19:26.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:19:26.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:19:26.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:19:26.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:26.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:27.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:27.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:27.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:27.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:27.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:27.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:27.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:27.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:27.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:27.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:27.953 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-18 zones={test-zone3, test-zone1} 2020-07-13T06:19:27.953 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-18 zones={test-zone3, test-zone2} 2020-07-13T06:19:28.243 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:19:28.254 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:19:28.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:19:28.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:19:28.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:19:28.256 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:19:28.266 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:19:28.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-13T06:19:28.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-13T06:19:28.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-13T06:19:28.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:28.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:28.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:19:28.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:28.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:28.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-13T06:19:28.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:28.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:28.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-13T06:19:29.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:29.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:29.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-13T06:19:29.302 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-18 zones={test-zone3, test-zone2} 2020-07-13T06:19:29.303 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2020-07-13T06:19:30.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:30.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:30.128 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:30.165 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:30.166 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:30.347 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=587bd98b-69d2-49ba-9dbd-2be72177aade 2020-07-13T06:19:30.376 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621170.123767_290.1","last_update":"2020-07-13T06:19:30.123767Z"},{"marker":"1_1594620933.443793_144.1","last_update":"2020-07-13T06:15:33.443793Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:19:30.377 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:19:30.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:30.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:30.379 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:30.419 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:30.420 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:30.643 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621170.123767_290.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:19:30.123767Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:19:30.644 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:19:30.644 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:19:30.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:30.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:30.646 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:30.688 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:30.688 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:30.918 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621170.123767_290.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:19:30.123767Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:19:30.918 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:19:30.919 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:19:30.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:30.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:30.921 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:30.960 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:30.961 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:31.233 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621170.123767_290.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:19:30.123767Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:19:31.234 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:19:31.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:31.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:31.600 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:31.644 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:31.645 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:31.849 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.41.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:31.850 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-19 2020-07-13T06:19:31.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:31.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:31.852 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:31.905 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:31.905 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:32.135 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000002.41.10","timestamp":"2020-07-13T06:19:31.766094Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:32.136 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-19 2020-07-13T06:19:32.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:32.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:32.138 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:32.178 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:32.179 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:32.377 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.45.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:32.378 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-19 2020-07-13T06:19:32.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:32.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:32.379 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:32.465 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:32.466 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:32.691 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000002.41.10","timestamp":"2020-07-13T06:19:31.761282Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:32.692 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone3 behind zone test-zone1: shard=5 master=00000000004.45.9 target=00000000002.41.10 2020-07-13T06:19:37.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:37.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:37.696 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:37.742 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:37.743 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:37.983 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000002.41.10","timestamp":"2020-07-13T06:19:31.761282Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:37.984 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone3 behind zone test-zone1: shard=5 master=00000000004.45.9 target=00000000002.41.10 2020-07-13T06:19:42.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:42.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:42.987 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:43.031 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:43.031 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:43.271 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000002.41.10","timestamp":"2020-07-13T06:19:31.761282Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:43.272 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone3 behind zone test-zone1: shard=5 master=00000000004.45.9 target=00000000002.41.10 2020-07-13T06:19:48.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:48.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:48.286 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:48.332 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:48.333 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:48.561 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000004.45.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:48.562 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-19 2020-07-13T06:19:48.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:48.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:48.564 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:48.643 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:48.643 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:48.867 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.47.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:48.868 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-19 2020-07-13T06:19:48.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:48.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:48.870 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:48.923 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:48.924 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:49.202 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.47.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:49.202 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-19 2020-07-13T06:19:49.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:49.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:49.204 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:49.260 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:49.261 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:49.499 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:49.500 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-19 2020-07-13T06:19:49.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:49.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:49.502 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:49.550 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:49.550 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:49.785 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:49.786 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-19 2020-07-13T06:19:49.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:49.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:49.788 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:49.833 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:49.834 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:50.060 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:50.061 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-19 2020-07-13T06:19:50.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:50.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:50.065 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:50.111 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:50.111 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:50.356 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:50.357 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-19 2020-07-13T06:19:50.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:50.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:50.359 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:50.436 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:50.437 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:50.630 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:19:50.631 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-19 2020-07-13T06:19:50.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:50.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:50.632 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:50.716 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:50.716 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:50.960 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000003.29.9","timestamp":"2020-07-13T06:19:32.121225Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:50.960 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone1 behind zone test-zone3: shard=5 master=00000000004.31.9 target=00000000003.29.9 2020-07-13T06:19:55.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:55.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:19:55.964 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:19:56.008 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:19:56.009 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:19:56.248 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000003.29.9","timestamp":"2020-07-13T06:19:32.121225Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:19:56.249 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone1 behind zone test-zone3: shard=5 master=00000000004.31.9 target=00000000003.29.9 2020-07-13T06:20:01.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:01.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:01.251 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:01.296 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:01.296 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:01.548 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000003.29.9","timestamp":"2020-07-13T06:19:32.121225Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:20:01.548 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone1 behind zone test-zone3: shard=5 master=00000000004.31.9 target=00000000003.29.9 2020-07-13T06:20:06.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:06.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:06.552 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:06.600 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:06.601 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:06.827 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:20:06.828 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-19 2020-07-13T06:20:06.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:06.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:06.830 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:06.873 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:06.873 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:07.092 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:20:07.093 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-19 2020-07-13T06:20:07.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:07.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:07.095 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:07.143 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:07.143 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:07.394 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:20:07.395 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-19 2020-07-13T06:20:07.395 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-19 zones={test-zone2, test-zone1} 2020-07-13T06:20:08.374 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:20:08.383 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:20:08.384 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:20:08.435 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:20:08.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:20:08.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:20:08.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:20:08.643 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-19 zones={test-zone2, test-zone1} 2020-07-13T06:20:08.644 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-19 zones={test-zone3, test-zone1} 2020-07-13T06:20:08.694 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:20:08.742 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:20:08.743 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:20:08.751 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:20:08.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:20:08.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:20:08.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:20:08.998 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-19 zones={test-zone3, test-zone1} 2020-07-13T06:20:08.999 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-19 zones={test-zone3, test-zone2} 2020-07-13T06:20:09.009 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:20:09.058 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:20:09.059 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:20:09.068 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:20:09.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:20:09.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:20:09.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:20:09.286 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-19 zones={test-zone3, test-zone2} 2020-07-13T06:20:10.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:10.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:10.812 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:10.852 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:10.853 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:11.084 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.47.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000028.88.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:20:11.085 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-19 2020-07-13T06:20:11.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:11.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:11.087 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:11.134 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:11.135 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:11.380 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000005.47.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"1-7GalX9EEsPPBii-4gazVhYWf-geka","ns":""},"count":1},"inc_marker":{"position":"00000000004.30.11","timestamp":"2020-07-13T06:20:09.458614Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:20:11.381 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone2 behind zone test-zone1: shard=8 master=00000000028.88.11 target=00000000004.30.11 2020-07-13T06:20:16.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:16.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:16.384 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:16.430 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:16.430 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:16.683 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000005.47.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"1-7GalX9EEsPPBii-4gazVhYWf-geka","ns":""},"count":1},"inc_marker":{"position":"00000000004.30.11","timestamp":"2020-07-13T06:20:09.458614Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:20:16.684 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone2 behind zone test-zone1: shard=8 master=00000000028.88.11 target=00000000004.30.11 2020-07-13T06:20:21.686 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:21.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:21.688 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:21.735 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:21.736 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:22.011 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000005.47.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"1-7GalX9EEsPPBii-4gazVhYWf-geka","ns":""},"count":1},"inc_marker":{"position":"00000000004.30.11","timestamp":"2020-07-13T06:20:09.458614Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:20:22.012 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone2 behind zone test-zone1: shard=8 master=00000000028.88.11 target=00000000004.30.11 2020-07-13T06:20:27.014 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:27.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:27.017 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:27.066 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:27.069 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:27.307 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000005.47.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"1-7GalX9EEsPPBii-4gazVhYWf-geka","ns":""},"count":1},"inc_marker":{"position":"00000000044.131.9","timestamp":"2020-07-13T06:20:10.766199Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:20:27.308 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-19 2020-07-13T06:20:27.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:27.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:27.310 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:27.352 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:27.353 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:27.564 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.47.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000044.131.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:20:27.565 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-19 2020-07-13T06:20:27.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:27.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:27.568 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:27.649 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:27.649 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:27.887 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000005.47.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"1-7GalX9EEsPPBii-4gazVhYWf-geka","ns":""},"count":1},"inc_marker":{"position":"00000000044.131.9","timestamp":"2020-07-13T06:20:10.766199Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:20:27.888 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-19 2020-07-13T06:20:27.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:27.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:27.891 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:27.971 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:27.972 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:28.194 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.47.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000044.131.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:20:28.207 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-19 2020-07-13T06:20:28.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:28.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:28.209 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:28.269 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:28.270 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:28.536 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.47.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000033.104.5","timestamp":"2020-07-13T06:20:09.961445Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:20:28.537 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone4 behind zone test-zone1: shard=8 master=00000000044.131.9 target=00000000033.104.5 2020-07-13T06:20:33.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:33.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:33.541 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:33.602 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:33.602 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:33.870 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.47.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000033.104.5","timestamp":"2020-07-13T06:20:09.961445Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:20:33.871 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone4 behind zone test-zone1: shard=8 master=00000000044.131.9 target=00000000033.104.5 2020-07-13T06:20:38.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:38.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:38.874 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:38.920 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:38.920 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:39.191 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.47.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000033.104.5","timestamp":"2020-07-13T06:20:09.961445Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:20:39.191 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone4 behind zone test-zone1: shard=8 master=00000000044.131.9 target=00000000033.104.5 2020-07-13T06:20:44.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:44.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:44.195 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:44.240 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:44.240 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:44.514 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.47.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000033.104.5","timestamp":"2020-07-13T06:20:09.961445Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:20:44.515 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone4 behind zone test-zone1: shard=8 master=00000000044.131.9 target=00000000033.104.5 2020-07-13T06:20:49.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:49.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:49.519 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:49.567 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:49.568 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:49.841 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.47.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000044.131.9","timestamp":"2020-07-13T06:20:10.766199Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:20:49.842 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-19 2020-07-13T06:20:49.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:49.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:49.844 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:49.884 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:49.884 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:50.108 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000047.126.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:20:50.108 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-19 2020-07-13T06:20:50.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:50.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:50.110 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:50.157 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:50.158 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:50.394 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"E8w36yuX-0.2h1YVD0852B5OLXXUrs8","ns":""},"count":1},"inc_marker":{"position":"00000000046.124.9","timestamp":"2020-07-13T06:20:10.766199Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:20:50.394 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone1 behind zone test-zone2: shard=8 master=00000000047.126.9 target=00000000046.124.9 2020-07-13T06:20:55.396 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:55.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:20:55.398 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:20:55.444 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:20:55.445 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:20:55.688 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"E8w36yuX-0.2h1YVD0852B5OLXXUrs8","ns":""},"count":1},"inc_marker":{"position":"00000000046.124.9","timestamp":"2020-07-13T06:20:10.766199Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:20:55.689 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone1 behind zone test-zone2: shard=8 master=00000000047.126.9 target=00000000046.124.9 2020-07-13T06:21:00.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:00.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:00.692 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:00.735 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:00.736 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:00.985 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"E8w36yuX-0.2h1YVD0852B5OLXXUrs8","ns":""},"count":1},"inc_marker":{"position":"00000000046.124.9","timestamp":"2020-07-13T06:20:10.766199Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:21:00.986 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone1 behind zone test-zone2: shard=8 master=00000000047.126.9 target=00000000046.124.9 2020-07-13T06:21:05.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:05.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:05.989 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:06.034 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:06.035 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:06.250 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"E8w36yuX-0.2h1YVD0852B5OLXXUrs8","ns":""},"count":1},"inc_marker":{"position":"00000000046.124.9","timestamp":"2020-07-13T06:20:10.766199Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:21:06.251 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone1 behind zone test-zone2: shard=8 master=00000000047.126.9 target=00000000046.124.9 2020-07-13T06:21:11.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:11.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:11.254 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:11.299 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:11.299 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:11.540 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"E8w36yuX-0.2h1YVD0852B5OLXXUrs8","ns":""},"count":1},"inc_marker":{"position":"00000000046.124.9","timestamp":"2020-07-13T06:20:10.766199Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:21:11.541 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone1 behind zone test-zone2: shard=8 master=00000000047.126.9 target=00000000046.124.9 2020-07-13T06:21:16.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:16.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:16.544 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:16.591 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:16.592 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:16.824 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"E8w36yuX-0.2h1YVD0852B5OLXXUrs8","ns":""},"count":1},"inc_marker":{"position":"00000000046.124.9","timestamp":"2020-07-13T06:20:10.766199Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:21:16.824 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone1 behind zone test-zone2: shard=8 master=00000000047.126.9 target=00000000046.124.9 2020-07-13T06:21:21.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:21.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:21.828 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:21.874 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:21.875 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:22.109 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"E8w36yuX-0.2h1YVD0852B5OLXXUrs8","ns":""},"count":1},"inc_marker":{"position":"00000000046.124.9","timestamp":"2020-07-13T06:20:10.766199Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:21:22.110 WARNING:rgw_multi.tests:bucket cngrzz-19 zone test-zone1 behind zone test-zone2: shard=8 master=00000000047.126.9 target=00000000046.124.9 2020-07-13T06:21:27.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:27.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:27.113 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:27.159 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:27.159 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:27.405 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"E8w36yuX-0.2h1YVD0852B5OLXXUrs8","ns":""},"count":1},"inc_marker":{"position":"00000000047.126.9","timestamp":"2020-07-13T06:20:10.661985Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:21:27.406 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-19 2020-07-13T06:21:27.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:27.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:27.408 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:27.449 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:27.450 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:27.668 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000047.126.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:21:27.668 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-19 2020-07-13T06:21:27.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:27.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:27.670 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:27.715 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:27.716 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:27.962 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"E8w36yuX-0.2h1YVD0852B5OLXXUrs8","ns":""},"count":1},"inc_marker":{"position":"00000000047.126.9","timestamp":"2020-07-13T06:20:10.661985Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:21:27.962 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-19 2020-07-13T06:21:27.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:27.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:27.964 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:28.003 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:28.004 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:28.216 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000051.129.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:21:28.217 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-19 2020-07-13T06:21:28.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:28.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:28.219 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:28.304 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:28.304 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:28.553 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"gqCHNDNRhWyd5WX5eX24rPFv4G4Vnii","ns":""},"count":3},"inc_marker":{"position":"00000000051.129.9","timestamp":"2020-07-13T06:20:10.661985Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:21:28.554 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-19 2020-07-13T06:21:28.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:28.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:28.556 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:28.600 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:28.600 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:28.818 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000051.129.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:21:28.818 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-19 2020-07-13T06:21:28.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:28.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:28.820 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:28.866 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:28.866 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:29.110 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"sA-taXd4uBfOjKAYAb-DD0N-ppEuOUf","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9","timestamp":"2020-07-13T06:19:31.551232Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"gqCHNDNRhWyd5WX5eX24rPFv4G4Vnii","ns":""},"count":3},"inc_marker":{"position":"00000000051.129.9","timestamp":"2020-07-13T06:20:10.661985Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:21:29.111 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-19 2020-07-13T06:21:29.111 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-19 zones={test-zone2, test-zone1} 2020-07-13T06:21:29.128 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:21:29.146 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:21:29.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.161 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:21:29.174 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:21:29.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.186 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.186 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:29.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:21:29.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:21:29.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:21:29.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:29.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:29.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:29.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:29.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:29.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:29.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:30.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:30.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:30.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:30.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:30.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:30.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:30.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:30.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:30.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:30.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:31.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:31.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:31.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:31.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:31.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:31.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:31.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:31.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:31.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:31.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:32.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:32.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:32.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:32.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:32.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:32.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:32.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:32.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:32.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:32.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:33.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:33.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:33.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:33.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:33.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:33.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:33.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:33.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:33.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:33.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:34.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:34.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:34.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:34.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:34.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:34.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:34.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:34.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:34.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:34.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:35.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:35.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:35.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:35.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:35.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:35.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:35.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:35.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:35.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:35.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:36.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:36.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:36.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:36.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:36.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:36.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:36.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:36.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:36.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:36.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:37.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:37.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:37.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:37.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:37.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:37.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:37.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:37.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:37.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:37.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:38.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:38.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:38.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:38.310 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-19 zones={test-zone2, test-zone1} 2020-07-13T06:21:38.310 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-19 zones={test-zone3, test-zone1} 2020-07-13T06:21:38.320 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:21:38.329 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:21:38.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.343 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:21:38.354 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:21:38.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:38.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:21:38.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:21:38.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:21:38.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:38.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:38.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:38.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:38.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:39.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:39.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:39.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:39.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:39.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:39.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:39.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:39.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:39.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:39.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:40.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:40.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:40.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:40.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:40.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:40.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:40.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:40.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:40.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:40.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:40.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:41.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:41.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:41.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:41.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:41.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:41.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:41.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:41.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:41.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:41.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:42.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:42.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:42.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:42.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:42.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:42.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:42.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:42.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:42.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:42.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:42.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:43.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:43.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:43.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:43.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:43.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:43.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:43.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:43.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:43.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:43.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:44.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:44.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:44.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:44.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:44.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:44.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:44.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:44.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:44.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:44.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:45.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:45.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:45.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:45.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:45.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:45.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:45.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:45.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:45.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:45.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:46.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:46.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:46.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:46.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:46.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:46.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:46.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:46.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:46.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:46.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:47.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:47.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:47.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:47.363 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-19 zones={test-zone3, test-zone1} 2020-07-13T06:21:47.364 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-19 zones={test-zone3, test-zone2} 2020-07-13T06:21:48.035 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:21:48.055 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:21:48.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.073 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.073 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.083 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:21:48.096 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-13T06:21:48.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:21:48.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:21:48.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:21:48.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-13T06:21:48.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:48.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:48.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:48.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:48.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:48.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:48.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:48.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:49.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:49.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:49.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:49.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:49.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:49.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:49.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:49.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:49.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:49.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:50.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:50.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:50.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:50.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:50.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:50.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:50.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:50.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:50.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:51.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:51.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:51.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:51.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:51.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:51.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:51.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:51.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:51.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:51.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:51.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:52.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:52.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:52.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:52.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:52.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:52.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:52.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:52.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:52.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:52.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:53.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:53.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:53.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:53.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:53.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:53.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:53.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:53.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:53.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:54.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:54.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:54.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:54.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:54.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:54.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:54.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:54.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:54.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:54.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:54.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:55.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:55.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:55.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:55.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:55.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:55.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:55.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:55.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:55.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:56.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:56.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:56.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:56.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:56.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:56.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:56.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:56.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:56.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:56.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:57.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:57.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:57.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:21:57.365 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-19 zones={test-zone3, test-zone2} 2020-07-13T06:21:57.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:57.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:57.370 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:57.410 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:57.411 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:57.647 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621203.033096_83.1","last_update":"2020-07-13T06:20:03.033096Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1594621137.230195_96.1","last_update":"2020-07-13T06:18:57.230195Z"},{"marker":"1_1594621269.047196_221.1","last_update":"2020-07-13T06:21:09.047196Z"}] 2020-07-13T06:21:57.648 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-13T06:21:57.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:57.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:57.650 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:57.693 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:57.694 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:57.929 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15669168566042326328},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621203.033096_83.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:20:03.033096Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621269.047196_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:21:09.047196Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:21:57.930 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-13T06:21:57.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:57.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:57.933 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:57.975 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:57.976 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:58.223 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621203.033096_83.1","last_update":"2020-07-13T06:20:03.033096Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1594621137.230195_96.1","last_update":"2020-07-13T06:18:57.230195Z"},{"marker":"1_1594621269.047196_221.1","last_update":"2020-07-13T06:21:09.047196Z"}] 2020-07-13T06:21:58.223 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-13T06:21:58.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:58.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:58.225 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:58.308 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:58.309 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:58.549 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2389703693262437130},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621203.033096_83.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:20:03.033096Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621269.047196_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:21:09.047196Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:21:58.550 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-13T06:21:58.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:58.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:58.552 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:58.625 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:58.626 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:58.876 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621203.033096_83.1","last_update":"2020-07-13T06:20:03.033096Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1594621137.230195_96.1","last_update":"2020-07-13T06:18:57.230195Z"},{"marker":"1_1594621269.047196_221.1","last_update":"2020-07-13T06:21:09.047196Z"}] 2020-07-13T06:21:58.876 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-13T06:21:58.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:58.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:58.878 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:58.922 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:58.922 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:59.202 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12943611757838484443},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621203.033096_83.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:20:03.033096Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621269.047196_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:21:09.047196Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:21:59.202 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-13T06:21:59.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:59.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:59.205 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:59.281 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:59.282 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:21:59.513 INFO:teuthology.orchestra.run.smithi014.stdout:[{"marker":"1_1594621203.127419_61.1","last_update":"2020-07-13T06:20:03.127419Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1594621137.114488_103.3","last_update":"2020-07-13T06:18:57.114488Z"},{"marker":"1_1594621269.140816_168.1","last_update":"2020-07-13T06:21:09.140816Z"}] 2020-07-13T06:21:59.752 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-13T06:21:59.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:59.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:21:59.755 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:21:59.798 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:21:59.798 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:00.046 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6157611387314156630},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621203.127419_61.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:20:03.127419Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.114488_103.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.114488Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621269.140816_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:21:09.140816Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:22:00.047 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-13T06:22:00.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:00.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:00.049 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:00.092 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:00.093 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:00.333 INFO:teuthology.orchestra.run.smithi014.stdout:[{"marker":"1_1594621203.127419_61.1","last_update":"2020-07-13T06:20:03.127419Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1594621137.114488_103.3","last_update":"2020-07-13T06:18:57.114488Z"},{"marker":"1_1594621269.140816_168.1","last_update":"2020-07-13T06:21:09.140816Z"}] 2020-07-13T06:22:00.334 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-13T06:22:00.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:00.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:00.337 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:00.383 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:00.384 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:00.607 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3188359685164663541},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621203.127419_61.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:20:03.127419Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.114488_103.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.114488Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621269.140816_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:21:09.140816Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:22:00.608 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-13T06:22:00.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:00.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:00.609 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:00.690 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:00.690 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:00.922 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621202.985988_64.1","last_update":"2020-07-13T06:20:02.985988Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1594621137.234104_70.1","last_update":"2020-07-13T06:18:57.234104Z"},{"marker":"1_1594621269.000208_221.1","last_update":"2020-07-13T06:21:09.000208Z"}] 2020-07-13T06:22:00.923 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-13T06:22:00.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:00.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:00.925 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:01.008 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:01.009 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:01.249 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3540715737640754362},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621202.985988_64.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:20:02.985988Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.234104_70.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.234104Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621269.000208_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:21:09.000208Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:22:01.250 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-13T06:22:01.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:01.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:01.253 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:01.332 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:01.333 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:01.585 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621202.985988_64.1","last_update":"2020-07-13T06:20:02.985988Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1594621137.234104_70.1","last_update":"2020-07-13T06:18:57.234104Z"},{"marker":"1_1594621269.000208_221.1","last_update":"2020-07-13T06:21:09.000208Z"}] 2020-07-13T06:22:01.586 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-13T06:22:01.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:01.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:01.588 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:01.632 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:01.632 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:01.871 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11134568813707895163},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621202.985988_64.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:20:02.985988Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.234104_70.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.234104Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621269.000208_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:21:09.000208Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:22:01.871 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-13T06:22:01.872 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2020-07-13T06:22:02.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:02.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:02.073 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:02.112 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:02.112 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:02.322 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=587bd98b-69d2-49ba-9dbd-2be72177aade 2020-07-13T06:22:02.345 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621322.065336_352.1","last_update":"2020-07-13T06:22:02.065336Z"},{"marker":"1_1594620933.443793_144.1","last_update":"2020-07-13T06:15:33.443793Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:22:02.346 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:22:02.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:02.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:02.348 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:02.400 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:02.401 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:02.623 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621322.065336_352.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:22:02.065336Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:22:02.624 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:22:02.625 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:22:02.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:02.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:02.626 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:02.676 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:02.677 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:02.914 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621322.065336_352.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:22:02.065336Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:22:02.915 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:22:02.915 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:22:02.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:02.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:02.917 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:02.960 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:02.960 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:03.237 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621322.065336_352.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:22:02.065336Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:22:03.239 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:22:03.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:03.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:03.336 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:03.379 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:03.379 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:03.613 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.15.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:22:03.613 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-20 2020-07-13T06:22:03.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:03.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:03.616 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:03.657 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:03.658 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:03.885 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.5","timestamp":"2020-07-13T06:22:03.450115Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:03.886 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-20 2020-07-13T06:22:03.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:03.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:03.889 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:03.932 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:03.933 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:04.174 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.15.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:22:04.175 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-20 2020-07-13T06:22:04.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:04.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:04.178 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:04.263 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:04.263 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:04.529 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.5","timestamp":"2020-07-13T06:22:03.449276Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:04.530 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-20 2020-07-13T06:22:04.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:04.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:04.533 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:04.575 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:04.576 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:04.810 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.15.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:22:04.810 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-20 2020-07-13T06:22:04.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:04.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:04.812 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:04.861 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:04.862 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:05.136 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.15.5","timestamp":"2020-07-13T06:22:03.289235Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:05.137 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-20 2020-07-13T06:22:05.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:05.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:05.139 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:05.218 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:05.218 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:05.436 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.13.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:22:05.437 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-20 2020-07-13T06:22:05.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:05.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:05.438 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:05.484 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:05.485 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:05.730 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-07-13T06:22:03.717364Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:05.731 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-20 2020-07-13T06:22:05.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:05.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:05.733 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:05.778 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:05.778 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:06.026 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.13.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:22:06.027 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-20 2020-07-13T06:22:06.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:06.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:06.029 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:06.077 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:06.077 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:06.322 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-07-13T06:22:03.721256Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:06.323 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-20 2020-07-13T06:22:06.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:06.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:06.325 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:06.406 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:06.407 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:06.638 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.17.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:22:06.639 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-20 2020-07-13T06:22:06.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:06.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:06.642 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:06.725 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:06.725 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:06.970 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.5","timestamp":"2020-07-13T06:22:03.809247Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:06.971 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-20 2020-07-13T06:22:06.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:06.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:06.973 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:07.055 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:07.056 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:07.291 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.17.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:22:07.292 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-20 2020-07-13T06:22:07.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:07.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:07.294 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:07.337 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:07.337 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:07.601 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.5","timestamp":"2020-07-13T06:22:03.814059Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:07.602 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-20 2020-07-13T06:22:07.602 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-20 zones={test-zone2, test-zone1} 2020-07-13T06:22:08.388 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:22:08.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:22:08.439 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:22:08.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:22:08.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:22:08.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:22:08.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:22:08.677 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-20 zones={test-zone2, test-zone1} 2020-07-13T06:22:08.677 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-20 zones={test-zone3, test-zone1} 2020-07-13T06:22:08.866 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:22:08.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:22:08.925 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:22:08.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:22:09.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:22:09.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:22:09.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:22:09.180 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-20 zones={test-zone3, test-zone1} 2020-07-13T06:22:09.181 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-20 zones={test-zone3, test-zone2} 2020-07-13T06:22:09.193 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:22:09.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:22:09.243 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:22:09.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:22:09.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:22:09.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:22:09.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:22:09.474 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-20 zones={test-zone3, test-zone2} 2020-07-13T06:22:09.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:09.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:09.495 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:09.540 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:09.541 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:09.711 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=587bd98b-69d2-49ba-9dbd-2be72177aade 2020-07-13T06:22:09.750 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621329.487977_368.1","last_update":"2020-07-13T06:22:09.487977Z"},{"marker":"1_1594620933.443793_144.1","last_update":"2020-07-13T06:15:33.443793Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:22:09.750 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:22:09.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:09.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:09.752 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:09.794 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:09.794 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:10.024 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621329.487977_368.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:22:09.487977Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:22:10.025 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:22:10.025 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:22:10.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:10.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:10.027 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:10.067 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:10.068 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:10.289 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621329.487977_368.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:22:09.487977Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:22:10.290 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:22:10.290 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:22:10.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:10.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:10.292 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:10.332 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:10.333 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:10.604 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621329.487977_368.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:22:09.487977Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:22:10.605 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:22:11.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:11.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:11.321 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:11.362 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:11.363 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:11.596 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.18.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:22:11.597 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-20 2020-07-13T06:22:11.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:11.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:11.599 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:11.650 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:11.650 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:11.896 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.5","timestamp":"2020-07-13T06:22:03.450115Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:11.897 WARNING:rgw_multi.tests:bucket cngrzz-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.18.12 target=00000000001.15.5 2020-07-13T06:22:16.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:16.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:16.900 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:16.948 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:16.949 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:17.201 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.12","timestamp":"2020-07-13T06:22:11.274913Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:17.201 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-20 2020-07-13T06:22:17.202 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:17.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:17.203 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:17.246 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:17.247 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:17.485 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.18.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:22:17.485 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-20 2020-07-13T06:22:17.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:17.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:17.487 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:17.570 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:17.571 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:17.818 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.5","timestamp":"2020-07-13T06:22:03.449276Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:17.818 WARNING:rgw_multi.tests:bucket cngrzz-20 zone test-zone3 behind zone test-zone1: shard=8 master=00000000003.18.12 target=00000000001.15.5 2020-07-13T06:22:22.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:22.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:22.822 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:22.872 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:22.872 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:23.127 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.5","timestamp":"2020-07-13T06:22:03.449276Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:23.128 WARNING:rgw_multi.tests:bucket cngrzz-20 zone test-zone3 behind zone test-zone1: shard=8 master=00000000003.18.12 target=00000000001.15.5 2020-07-13T06:22:28.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:28.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:28.133 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:28.180 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:28.180 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:28.435 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.12","timestamp":"2020-07-13T06:22:11.274913Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:28.436 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-20 2020-07-13T06:22:28.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:28.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:28.438 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:28.478 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:28.479 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:28.700 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.18.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:22:28.701 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-20 2020-07-13T06:22:28.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:28.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:28.703 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:28.757 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:28.757 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:29.031 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.18.12","timestamp":"2020-07-13T06:22:11.274913Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:29.032 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-20 2020-07-13T06:22:29.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:29.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:29.033 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:29.074 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:29.074 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:29.316 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.17.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:22:29.316 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-20 2020-07-13T06:22:29.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:29.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:29.318 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:29.363 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:29.364 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:29.595 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-07-13T06:22:03.717364Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:29.596 WARNING:rgw_multi.tests:bucket cngrzz-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000003.17.12 target=00000000001.13.5 2020-07-13T06:22:34.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:34.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:34.599 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:34.644 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:34.644 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:34.887 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-07-13T06:22:03.717364Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:34.887 WARNING:rgw_multi.tests:bucket cngrzz-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000003.17.12 target=00000000001.13.5 2020-07-13T06:22:39.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:39.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:39.891 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:39.937 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:39.937 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:40.199 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-07-13T06:22:03.717364Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:40.199 WARNING:rgw_multi.tests:bucket cngrzz-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000003.17.12 target=00000000001.13.5 2020-07-13T06:22:45.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:45.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:45.203 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:45.250 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:45.250 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:45.487 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.9","timestamp":"2020-07-13T06:22:11.274913Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:45.488 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-20 2020-07-13T06:22:45.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:45.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:45.490 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:45.534 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:45.534 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:45.753 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.19.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:22:45.754 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-20 2020-07-13T06:22:45.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:45.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:45.755 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:45.800 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:45.800 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:46.048 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.9","timestamp":"2020-07-13T06:22:11.274913Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:46.049 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-20 2020-07-13T06:22:46.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:46.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:46.051 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:46.130 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:46.130 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:46.353 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.23.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:22:46.353 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-20 2020-07-13T06:22:46.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:46.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:46.355 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:46.437 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:46.438 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:46.659 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.21.12","timestamp":"2020-07-13T06:22:11.274913Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:46.659 WARNING:rgw_multi.tests:bucket cngrzz-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000004.23.9 target=00000000003.21.12 2020-07-13T06:22:51.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:51.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:51.663 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:51.707 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:51.708 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:51.953 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.21.12","timestamp":"2020-07-13T06:22:11.274913Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:51.954 WARNING:rgw_multi.tests:bucket cngrzz-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000004.23.9 target=00000000003.21.12 2020-07-13T06:22:56.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:56.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:22:56.958 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:22:57.003 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:22:57.003 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:22:57.252 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.21.12","timestamp":"2020-07-13T06:22:11.274913Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:22:57.253 WARNING:rgw_multi.tests:bucket cngrzz-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000004.23.9 target=00000000003.21.12 2020-07-13T06:23:02.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:02.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:02.257 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:02.304 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:02.304 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:02.546 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.21.12","timestamp":"2020-07-13T06:22:11.274913Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:02.547 WARNING:rgw_multi.tests:bucket cngrzz-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000004.23.9 target=00000000003.21.12 2020-07-13T06:23:07.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:07.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:07.549 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:07.593 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:07.594 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:07.839 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.9","timestamp":"2020-07-13T06:22:11.274913Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:07.839 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-20 2020-07-13T06:23:07.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:07.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:07.842 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:07.927 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:07.927 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:08.164 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.23.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:23:08.164 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-20 2020-07-13T06:23:08.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:08.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:08.166 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:08.211 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:08.212 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:08.459 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.9","timestamp":"2020-07-13T06:22:11.274913Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:08.460 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-20 2020-07-13T06:23:08.461 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-20 zones={test-zone2, test-zone1} 2020-07-13T06:23:09.299 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:23:09.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:09.310 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:09.311 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:23:09.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:09.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:09.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:09.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:09.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:09.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:09.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:09.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:09.742 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-20 zones={test-zone2, test-zone1} 2020-07-13T06:23:10.009 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-20 zones={test-zone3, test-zone1} 2020-07-13T06:23:10.027 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:23:10.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:10.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:10.082 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:23:10.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:10.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:10.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:10.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:10.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:10.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:10.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:10.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:10.468 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-20 zones={test-zone3, test-zone1} 2020-07-13T06:23:10.469 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-20 zones={test-zone3, test-zone2} 2020-07-13T06:23:10.480 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:23:10.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:10.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:10.535 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:23:10.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:10.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:10.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:10.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:10.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:10.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:10.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:10.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:10.910 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-20 zones={test-zone3, test-zone2} 2020-07-13T06:23:10.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:10.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:10.930 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:10.971 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:10.971 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:11.152 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=587bd98b-69d2-49ba-9dbd-2be72177aade 2020-07-13T06:23:11.191 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621390.921870_405.1","last_update":"2020-07-13T06:23:10.921870Z"},{"marker":"1_1594620933.443793_144.1","last_update":"2020-07-13T06:15:33.443793Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:23:11.191 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:23:11.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:11.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:11.193 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:11.233 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:11.233 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:11.461 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621390.921870_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:23:10.921870Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:23:11.462 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:23:11.463 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:23:11.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:11.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:11.465 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:11.509 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:11.509 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:11.721 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621390.921870_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:23:10.921870Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:23:11.722 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:23:11.722 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:23:11.723 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:11.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:11.725 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:11.765 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:11.766 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:12.035 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621390.921870_405.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:23:10.921870Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:23:12.036 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:23:12.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:12.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:12.677 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:12.720 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:12.721 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:12.934 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.22.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:23:12.935 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-20 2020-07-13T06:23:12.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:12.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:12.938 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:12.984 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:12.985 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:13.212 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000005.22.17","timestamp":"2020-07-13T06:23:12.630111Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:13.212 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-20 2020-07-13T06:23:13.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:13.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:13.214 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:13.257 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:13.258 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:13.498 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.22.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:23:13.499 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-20 2020-07-13T06:23:13.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:13.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:13.500 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:13.582 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:13.583 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:13.822 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000005.22.17","timestamp":"2020-07-13T06:23:12.630111Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:13.822 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-20 2020-07-13T06:23:13.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:13.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:13.824 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:13.909 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:13.910 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:14.136 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.22.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:23:14.136 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-20 2020-07-13T06:23:14.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:14.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:14.138 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:14.185 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:14.186 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:14.455 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.22.17","timestamp":"2020-07-13T06:23:12.630111Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:14.456 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-20 2020-07-13T06:23:14.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:14.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:14.458 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:14.497 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:14.497 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:14.723 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000006.24.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:23:14.724 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-20 2020-07-13T06:23:14.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:14.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:14.725 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:14.785 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:14.786 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:15.022 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.9","timestamp":"2020-07-13T06:22:11.274913Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:15.023 WARNING:rgw_multi.tests:bucket cngrzz-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000006.24.17 target=00000000004.19.9 2020-07-13T06:23:20.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:20.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:20.026 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:20.072 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:20.073 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:20.322 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.9","timestamp":"2020-07-13T06:22:11.274913Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:20.323 WARNING:rgw_multi.tests:bucket cngrzz-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000006.24.17 target=00000000004.19.9 2020-07-13T06:23:25.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:25.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:25.325 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:25.374 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:25.375 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:25.596 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.26.14","timestamp":"2020-07-13T06:23:12.630111Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:25.596 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-20 2020-07-13T06:23:25.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:25.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:25.598 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:25.645 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:25.645 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:25.877 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.26.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:23:25.877 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-20 2020-07-13T06:23:25.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:25.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:25.879 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:25.925 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:25.926 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:26.164 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.26.14","timestamp":"2020-07-13T06:23:12.630111Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:26.165 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-20 2020-07-13T06:23:26.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:26.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:26.166 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:26.247 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:26.248 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:26.470 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.30.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:23:26.471 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-20 2020-07-13T06:23:26.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:26.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:26.473 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:26.563 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:26.564 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:26.808 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.28.17","timestamp":"2020-07-13T06:23:12.630111Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:26.808 WARNING:rgw_multi.tests:bucket cngrzz-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000007.30.14 target=00000000006.28.17 2020-07-13T06:23:31.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:31.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:31.811 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:31.862 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:31.862 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:32.119 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.28.17","timestamp":"2020-07-13T06:23:12.630111Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:32.120 WARNING:rgw_multi.tests:bucket cngrzz-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000007.30.14 target=00000000006.28.17 2020-07-13T06:23:37.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:37.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:37.123 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:37.168 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:37.168 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:37.421 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.28.17","timestamp":"2020-07-13T06:23:12.630111Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:37.422 WARNING:rgw_multi.tests:bucket cngrzz-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000007.30.14 target=00000000006.28.17 2020-07-13T06:23:42.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:42.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:42.425 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:42.471 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:42.471 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:42.719 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.28.17","timestamp":"2020-07-13T06:23:12.630111Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:42.720 WARNING:rgw_multi.tests:bucket cngrzz-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000007.30.14 target=00000000006.28.17 2020-07-13T06:23:47.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:47.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:47.723 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:47.770 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:47.770 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:48.009 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.30.14","timestamp":"2020-07-13T06:23:12.630111Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:48.010 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-20 2020-07-13T06:23:48.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:48.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:48.012 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:48.106 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:48.106 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:48.328 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.30.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:23:48.328 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-20 2020-07-13T06:23:48.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:48.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:48.330 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:48.384 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:48.384 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:48.629 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.30.14","timestamp":"2020-07-13T06:23:12.630111Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:48.630 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-20 2020-07-13T06:23:48.631 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-20 zones={test-zone2, test-zone1} 2020-07-13T06:23:49.648 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:23:49.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:49.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:49.657 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:23:49.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:49.708 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:49.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:49.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:49.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:49.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:49.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:50.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:50.078 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-20 zones={test-zone2, test-zone1} 2020-07-13T06:23:50.079 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-20 zones={test-zone3, test-zone1} 2020-07-13T06:23:50.785 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:23:50.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:50.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:50.836 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:23:50.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:50.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:50.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:50.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:50.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:51.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:51.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:51.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:51.203 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-20 zones={test-zone3, test-zone1} 2020-07-13T06:23:51.204 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-20 zones={test-zone3, test-zone2} 2020-07-13T06:23:51.879 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:23:51.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:51.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:51.892 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:23:51.942 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:51.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:23:52.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:52.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:52.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:52.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:52.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:52.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:23:52.300 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-20 zones={test-zone3, test-zone2} 2020-07-13T06:23:52.301 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2020-07-13T06:23:52.302 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-21 2020-07-13T06:23:53.334 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-22 2020-07-13T06:23:53.423 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-23 2020-07-13T06:23:54.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:54.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:54.315 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:54.355 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:54.355 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:54.537 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=587bd98b-69d2-49ba-9dbd-2be72177aade 2020-07-13T06:23:54.571 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621434.303397_449.1","last_update":"2020-07-13T06:23:54.303397Z"},{"marker":"1_1594620933.443793_144.1","last_update":"2020-07-13T06:15:33.443793Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:23:54.572 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:23:54.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:54.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:54.574 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:54.615 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:54.616 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:54.843 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621434.303397_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:23:54.303397Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:23:54.844 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:23:54.845 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:23:54.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:54.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:54.847 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:54.887 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:54.887 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:55.111 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621434.303397_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:23:54.303397Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:23:55.112 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:23:55.113 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:23:55.113 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:55.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:55.115 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:55.157 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:55.157 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:55.428 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621434.303397_449.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:23:54.303397Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:23:55.429 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:23:56.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:56.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:56.644 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:56.691 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:56.691 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:56.885 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.142.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:23:56.886 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-21 2020-07-13T06:23:56.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:56.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:56.888 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:56.935 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:56.935 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:57.182 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"BiQSOFmCyBkF1yJjM1hAz6Q6syKHGxs","ns":""},"count":2},"inc_marker":{"position":"00000000003.142.11","timestamp":"2020-07-13T06:23:55.914109Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:57.183 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-21 2020-07-13T06:23:57.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:57.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:57.185 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:57.229 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:57.229 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:57.461 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.142.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:23:57.461 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-21 2020-07-13T06:23:57.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:57.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:57.463 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:57.546 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:57.546 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:57.787 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"BiQSOFmCyBkF1yJjM1hAz6Q6syKHGxs","ns":""},"count":2},"inc_marker":{"position":"00000000003.142.11","timestamp":"2020-07-13T06:23:55.913314Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:57.787 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-21 2020-07-13T06:23:57.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:57.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:57.789 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:57.871 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:57.871 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:58.115 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.142.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:23:58.116 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-21 2020-07-13T06:23:58.116 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:58.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:58.118 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:58.161 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:58.161 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:58.440 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.142.11","timestamp":"2020-07-13T06:23:55.790351Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:58.441 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-21 2020-07-13T06:23:58.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:58.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:58.442 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:58.524 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:58.525 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:58.744 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.137.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:23:58.744 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-21 2020-07-13T06:23:58.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:58.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:58.746 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:58.791 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:58.791 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:59.041 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"BiQSOFmCyBkF1yJjM1hAz6Q6syKHGxs","ns":""},"count":2},"inc_marker":{"position":"00000000002.137.5","timestamp":"2020-07-13T06:23:56.185335Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:59.042 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-21 2020-07-13T06:23:59.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:59.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:59.044 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:59.087 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:59.088 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:59.324 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.137.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:23:59.324 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-21 2020-07-13T06:23:59.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:59.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:59.326 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:59.370 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:59.370 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:59.599 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"BiQSOFmCyBkF1yJjM1hAz6Q6syKHGxs","ns":""},"count":2},"inc_marker":{"position":"00000000002.137.5","timestamp":"2020-07-13T06:23:56.185340Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:23:59.600 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-21 2020-07-13T06:23:59.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:59.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:59.601 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:23:59.685 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:23:59.686 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:23:59.933 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.141.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:23:59.934 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-21 2020-07-13T06:23:59.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:59.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:23:59.936 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:00.022 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:00.022 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:00.236 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"BiQSOFmCyBkF1yJjM1hAz6Q6syKHGxs","ns":""},"count":2},"inc_marker":{"position":"00000000002.141.5","timestamp":"2020-07-13T06:23:55.739566Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:00.237 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-21 2020-07-13T06:24:00.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:00.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:00.239 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:00.279 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:00.280 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:00.511 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.141.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:00.511 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-21 2020-07-13T06:24:00.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:00.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:00.513 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:00.558 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:00.558 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:00.808 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"BiQSOFmCyBkF1yJjM1hAz6Q6syKHGxs","ns":""},"count":2},"inc_marker":{"position":"00000000002.141.5","timestamp":"2020-07-13T06:23:56.074114Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:00.809 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-21 2020-07-13T06:24:00.809 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-21 zones={test-zone2, test-zone1} 2020-07-13T06:24:01.774 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:24:01.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:01.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:01.784 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:24:01.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:01.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:01.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:01.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:01.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:01.996 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-21 zones={test-zone2, test-zone1} 2020-07-13T06:24:01.997 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-21 zones={test-zone3, test-zone1} 2020-07-13T06:24:02.017 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:24:02.066 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:02.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:02.067 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:24:02.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:02.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:02.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:02.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:02.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:02.266 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-21 zones={test-zone3, test-zone1} 2020-07-13T06:24:02.267 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-21 zones={test-zone3, test-zone2} 2020-07-13T06:24:02.276 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:24:02.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:02.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:02.328 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:24:02.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:02.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:02.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:02.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:02.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:02.493 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-21 zones={test-zone3, test-zone2} 2020-07-13T06:24:02.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:02.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:02.495 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:02.540 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:02.541 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:02.755 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.14.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:02.756 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-22 2020-07-13T06:24:02.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:02.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:02.758 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:02.803 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:02.804 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:03.061 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"nihrLIuzLwWssnubAO0zdHrGdIYr4eC","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.5","timestamp":"2020-07-13T06:23:56.714147Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:03.062 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-22 2020-07-13T06:24:03.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:03.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:03.064 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:03.109 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:03.110 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:03.351 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.14.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:03.352 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-22 2020-07-13T06:24:03.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:03.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:03.354 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:03.444 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:03.445 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:03.697 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"nihrLIuzLwWssnubAO0zdHrGdIYr4eC","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.5","timestamp":"2020-07-13T06:23:56.709217Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:03.698 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-22 2020-07-13T06:24:03.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:03.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:03.700 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:03.740 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:03.741 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:03.972 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.14.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:03.973 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-22 2020-07-13T06:24:03.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:03.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:03.975 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:04.024 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:04.025 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:04.293 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.14.5","timestamp":"2020-07-13T06:23:56.478375Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:04.294 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-22 2020-07-13T06:24:04.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:04.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:04.296 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:04.376 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:04.376 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:04.606 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.17.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:04.607 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-22 2020-07-13T06:24:04.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:04.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:04.608 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:04.650 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:04.650 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:04.917 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"nihrLIuzLwWssnubAO0zdHrGdIYr4eC","ns":""},"count":1},"inc_marker":{"position":"00000000003.17.11","timestamp":"2020-07-13T06:23:56.534647Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:04.918 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-22 2020-07-13T06:24:04.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:04.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:04.920 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:04.961 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:04.962 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:05.196 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.17.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:05.196 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-22 2020-07-13T06:24:05.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:05.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:05.198 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:05.243 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:05.243 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:05.489 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"nihrLIuzLwWssnubAO0zdHrGdIYr4eC","ns":""},"count":1},"inc_marker":{"position":"00000000003.17.11","timestamp":"2020-07-13T06:23:56.534647Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:05.490 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-22 2020-07-13T06:24:05.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:05.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:05.491 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:05.531 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:05.532 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:05.770 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.18.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:05.771 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-22 2020-07-13T06:24:05.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:05.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:05.773 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:05.860 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:05.861 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:06.116 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"nihrLIuzLwWssnubAO0zdHrGdIYr4eC","ns":""},"count":1},"inc_marker":{"position":"00000000002.18.5","timestamp":"2020-07-13T06:23:56.478375Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:06.117 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-22 2020-07-13T06:24:06.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:06.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:06.119 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:06.159 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:06.160 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:06.394 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.18.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:06.394 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-22 2020-07-13T06:24:06.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:06.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:06.396 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:06.441 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:06.441 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:06.684 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"nihrLIuzLwWssnubAO0zdHrGdIYr4eC","ns":""},"count":1},"inc_marker":{"position":"00000000002.18.5","timestamp":"2020-07-13T06:23:56.478375Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:06.685 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-22 2020-07-13T06:24:06.686 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-22 zones={test-zone2, test-zone1} 2020-07-13T06:24:07.790 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:24:07.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:07.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:07.800 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:24:07.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:07.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:07.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:07.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:07.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:08.016 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-22 zones={test-zone2, test-zone1} 2020-07-13T06:24:08.016 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-22 zones={test-zone3, test-zone1} 2020-07-13T06:24:08.499 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:24:08.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:08.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:08.547 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:24:08.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:08.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:08.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:08.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:08.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:08.710 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-22 zones={test-zone3, test-zone1} 2020-07-13T06:24:08.711 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-22 zones={test-zone3, test-zone2} 2020-07-13T06:24:08.719 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:24:08.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:08.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:08.772 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:24:08.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:08.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:08.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:08.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:08.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:08.935 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-22 zones={test-zone3, test-zone2} 2020-07-13T06:24:08.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:08.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:08.938 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:08.984 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:08.984 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:09.236 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.38.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:09.237 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-23 2020-07-13T06:24:09.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:09.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:09.238 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:09.285 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:09.285 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:09.546 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"XwJa6s0uZ8BrRCcHog.9xoSGjb.EoZp","ns":""},"count":1},"inc_marker":{"position":"00000000002.38.5","timestamp":"2020-07-13T06:23:56.954143Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:09.547 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-23 2020-07-13T06:24:09.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:09.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:09.549 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:09.591 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:09.592 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:09.840 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.38.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:09.841 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-23 2020-07-13T06:24:09.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:09.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:09.843 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:09.929 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:09.930 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:10.176 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"XwJa6s0uZ8BrRCcHog.9xoSGjb.EoZp","ns":""},"count":1},"inc_marker":{"position":"00000000002.38.5","timestamp":"2020-07-13T06:23:56.957308Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:10.176 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-23 2020-07-13T06:24:10.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:10.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:10.178 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:10.224 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:10.224 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:10.460 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.38.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:10.460 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-23 2020-07-13T06:24:10.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:10.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:10.462 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:10.511 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:10.511 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:10.769 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.38.5","timestamp":"2020-07-13T06:23:56.576413Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:10.770 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-23 2020-07-13T06:24:10.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:10.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:10.772 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:10.810 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:10.810 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:11.052 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.40.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:11.053 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-23 2020-07-13T06:24:11.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:11.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:11.055 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:11.099 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:11.100 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:11.362 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"XwJa6s0uZ8BrRCcHog.9xoSGjb.EoZp","ns":""},"count":1},"inc_marker":{"position":"00000000002.40.5","timestamp":"2020-07-13T06:23:56.781210Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:11.363 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-23 2020-07-13T06:24:11.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:11.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:11.365 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:11.410 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:11.411 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:11.644 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.40.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:11.645 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-23 2020-07-13T06:24:11.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:11.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:11.647 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:11.695 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:11.695 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:11.940 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"XwJa6s0uZ8BrRCcHog.9xoSGjb.EoZp","ns":""},"count":1},"inc_marker":{"position":"00000000002.40.5","timestamp":"2020-07-13T06:23:56.777214Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:11.941 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-23 2020-07-13T06:24:11.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:11.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:11.942 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:12.023 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:12.024 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:12.257 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.56.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:12.257 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-23 2020-07-13T06:24:12.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:12.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:12.259 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:12.344 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:12.344 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:12.613 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"XwJa6s0uZ8BrRCcHog.9xoSGjb.EoZp","ns":""},"count":1},"inc_marker":{"position":"00000000003.56.11","timestamp":"2020-07-13T06:23:56.626901Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:12.613 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-23 2020-07-13T06:24:12.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:12.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:12.615 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:12.653 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:12.654 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:12.884 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.56.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:12.885 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-23 2020-07-13T06:24:12.886 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:12.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:12.887 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:12.937 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:12.938 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:13.186 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"XwJa6s0uZ8BrRCcHog.9xoSGjb.EoZp","ns":""},"count":1},"inc_marker":{"position":"00000000003.56.11","timestamp":"2020-07-13T06:23:56.626901Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:13.187 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-23 2020-07-13T06:24:13.188 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-23 zones={test-zone2, test-zone1} 2020-07-13T06:24:14.128 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:24:14.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:14.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:14.138 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:24:14.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:14.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:14.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:14.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:14.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:14.360 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-23 zones={test-zone2, test-zone1} 2020-07-13T06:24:14.360 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-23 zones={test-zone3, test-zone1} 2020-07-13T06:24:14.819 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:24:14.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:14.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:14.880 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:24:14.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:14.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:14.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:14.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:14.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:15.067 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-23 zones={test-zone3, test-zone1} 2020-07-13T06:24:15.068 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-23 zones={test-zone3, test-zone2} 2020-07-13T06:24:15.078 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:24:15.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:15.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:15.127 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:24:15.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:15.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:15.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:15.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:15.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:15.287 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-23 zones={test-zone3, test-zone2} 2020-07-13T06:24:15.288 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2020-07-13T06:24:15.288 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-24 2020-07-13T06:24:15.324 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-25 2020-07-13T06:24:15.380 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-26 2020-07-13T06:24:15.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:15.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:15.574 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:15.610 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:15.610 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:15.800 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=587bd98b-69d2-49ba-9dbd-2be72177aade 2020-07-13T06:24:15.839 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621455.551984_514.1","last_update":"2020-07-13T06:24:15.551984Z"},{"marker":"1_1594620933.443793_144.1","last_update":"2020-07-13T06:15:33.443793Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:24:15.840 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:24:15.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:15.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:15.842 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:15.884 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:15.885 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:16.147 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621455.551984_514.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:24:15.551984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:24:16.148 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:24:16.149 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:24:16.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:16.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:16.155 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:16.198 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:16.199 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:16.431 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621455.551984_514.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:24:15.551984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:24:16.432 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:24:16.432 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:24:16.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:16.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:16.434 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:16.479 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:16.479 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:16.750 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621455.551984_514.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:24:15.551984Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:24:16.751 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:24:18.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:18.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:18.677 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:18.718 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:18.719 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:18.949 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.41.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:18.949 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-24 2020-07-13T06:24:18.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:18.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:18.951 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:18.997 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:18.997 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:19.249 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.41.6","timestamp":"2020-07-13T06:24:17.251252Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:19.250 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-24 2020-07-13T06:24:19.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:19.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:19.252 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:19.293 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:19.293 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:19.544 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.41.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:19.544 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-24 2020-07-13T06:24:19.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:19.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:19.546 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:19.631 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:19.631 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:19.894 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.41.6","timestamp":"2020-07-13T06:24:17.251252Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:19.895 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-24 2020-07-13T06:24:19.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:19.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:19.898 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:19.941 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:19.941 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:20.197 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.41.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:20.198 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-24 2020-07-13T06:24:20.199 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:20.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:20.200 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:20.244 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:20.245 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:20.515 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.41.6","timestamp":"2020-07-13T06:24:17.251252Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:20.515 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-24 2020-07-13T06:24:20.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:20.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:20.517 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:20.604 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:20.604 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:20.822 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.28.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:20.823 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-24 2020-07-13T06:24:20.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:20.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:20.825 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:20.868 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:20.869 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:21.117 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.12","timestamp":"2020-07-13T06:24:17.601238Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:21.118 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-24 2020-07-13T06:24:21.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:21.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:21.120 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:21.161 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:21.161 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:21.402 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.28.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:21.403 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-24 2020-07-13T06:24:21.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:21.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:21.404 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:21.448 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:21.448 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:21.703 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.12","timestamp":"2020-07-13T06:24:17.593359Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:21.704 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-24 2020-07-13T06:24:21.704 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:21.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:21.706 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:21.785 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:21.786 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:22.028 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.30.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:22.029 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-24 2020-07-13T06:24:22.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:22.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:22.031 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:22.113 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:22.113 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:22.371 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.12","timestamp":"2020-07-13T06:24:17.481192Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:22.372 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-24 2020-07-13T06:24:22.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:22.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:22.374 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:22.454 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:22.454 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:22.682 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.30.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:22.683 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-24 2020-07-13T06:24:22.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:22.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:22.685 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:22.730 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:22.730 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:22.991 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.12","timestamp":"2020-07-13T06:24:17.482086Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:22.992 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-24 2020-07-13T06:24:22.993 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-24 zones={test-zone2, test-zone1} 2020-07-13T06:24:23.706 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:24:23.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:23.716 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:24:23.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:23.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:23.832 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-24 zones={test-zone2, test-zone1} 2020-07-13T06:24:23.833 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-24 zones={test-zone3, test-zone1} 2020-07-13T06:24:24.748 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:24:24.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:24.795 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:24:24.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:24.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:24.872 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-24 zones={test-zone3, test-zone1} 2020-07-13T06:24:24.873 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-24 zones={test-zone3, test-zone2} 2020-07-13T06:24:25.386 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:24:25.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:25.397 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:24:25.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:25.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:25.512 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-24 zones={test-zone3, test-zone2} 2020-07-13T06:24:25.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:25.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:25.514 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:25.558 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:25.559 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:25.799 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.28.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:25.800 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-25 2020-07-13T06:24:25.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:25.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:25.801 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:25.847 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:25.847 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:26.121 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.12","timestamp":"2020-07-13T06:24:18.126107Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:26.122 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-25 2020-07-13T06:24:26.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:26.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:26.124 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:26.165 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:26.165 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:26.414 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.28.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:26.414 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-25 2020-07-13T06:24:26.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:26.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:26.416 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:26.502 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:26.502 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:26.746 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.12","timestamp":"2020-07-13T06:24:18.121309Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:26.747 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-25 2020-07-13T06:24:26.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:26.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:26.749 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:26.831 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:26.832 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:27.111 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.28.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:27.112 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-25 2020-07-13T06:24:27.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:27.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:27.114 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:27.158 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:27.158 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:27.427 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.28.12","timestamp":"2020-07-13T06:24:17.822338Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:27.428 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-25 2020-07-13T06:24:27.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:27.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:27.429 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:27.512 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:27.513 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:27.748 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.81.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:27.749 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-25 2020-07-13T06:24:27.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:27.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:27.751 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:27.796 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:27.796 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:28.071 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.81.6","timestamp":"2020-07-13T06:24:17.993348Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:28.072 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-25 2020-07-13T06:24:28.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:28.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:28.073 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:28.116 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:28.116 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:28.371 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.81.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:28.372 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-25 2020-07-13T06:24:28.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:28.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:28.375 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:28.421 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:28.422 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:28.660 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.81.6","timestamp":"2020-07-13T06:24:17.993373Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:28.661 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-25 2020-07-13T06:24:28.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:28.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:28.663 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:28.743 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:28.744 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:28.990 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.34.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:28.990 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-25 2020-07-13T06:24:28.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:28.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:28.992 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:29.076 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:29.077 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:29.350 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.34.12","timestamp":"2020-07-13T06:24:18.281242Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:29.351 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-25 2020-07-13T06:24:29.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:29.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:29.353 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:29.434 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:29.434 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:29.652 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.34.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:29.653 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-25 2020-07-13T06:24:29.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:29.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:29.662 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:29.709 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:29.709 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:29.987 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.34.12","timestamp":"2020-07-13T06:24:18.282109Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:29.988 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-25 2020-07-13T06:24:29.988 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-25 zones={test-zone2, test-zone1} 2020-07-13T06:24:31.009 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:24:31.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:31.019 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:24:31.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:31.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:31.137 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-25 zones={test-zone2, test-zone1} 2020-07-13T06:24:31.138 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-25 zones={test-zone3, test-zone1} 2020-07-13T06:24:31.833 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:24:31.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:31.883 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:24:31.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:31.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:31.952 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-25 zones={test-zone3, test-zone1} 2020-07-13T06:24:31.953 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-25 zones={test-zone3, test-zone2} 2020-07-13T06:24:31.964 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:24:32.022 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:32.023 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:24:32.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:32.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:32.099 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-25 zones={test-zone3, test-zone2} 2020-07-13T06:24:32.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:32.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:32.101 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:32.146 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:32.146 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:32.386 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.13.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:32.386 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-26 2020-07-13T06:24:32.387 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:32.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:32.388 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:32.432 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:32.433 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:32.709 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.12","timestamp":"2020-07-13T06:24:18.926121Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:32.709 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-26 2020-07-13T06:24:32.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:32.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:32.711 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:32.759 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:32.760 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:32.996 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.13.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:32.996 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-26 2020-07-13T06:24:32.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:32.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:32.998 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:33.081 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:33.082 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:33.326 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.12","timestamp":"2020-07-13T06:24:18.925217Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:33.327 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-26 2020-07-13T06:24:33.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:33.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:33.330 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:33.410 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:33.410 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:33.632 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.13.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:33.632 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-26 2020-07-13T06:24:33.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:33.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:33.635 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:33.681 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:33.682 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:33.957 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.13.12","timestamp":"2020-07-13T06:24:18.648621Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:33.958 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-26 2020-07-13T06:24:33.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:33.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:33.960 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:34.040 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:34.041 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:34.281 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.15.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:34.282 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-26 2020-07-13T06:24:34.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:34.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:34.283 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:34.328 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:34.328 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:34.571 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.12","timestamp":"2020-07-13T06:24:18.989320Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:34.572 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-26 2020-07-13T06:24:34.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:34.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:34.574 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:34.615 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:34.616 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:34.851 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.15.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:34.852 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-26 2020-07-13T06:24:34.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:34.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:34.854 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:34.900 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:34.901 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:35.153 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.12","timestamp":"2020-07-13T06:24:18.989335Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:35.154 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-26 2020-07-13T06:24:35.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:35.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:35.156 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:35.198 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:35.198 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:35.452 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.25.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:35.453 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-26 2020-07-13T06:24:35.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:35.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:35.456 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:35.543 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:35.544 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:35.794 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.25.6","timestamp":"2020-07-13T06:24:18.598609Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:35.794 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-26 2020-07-13T06:24:35.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:35.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:35.796 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:35.838 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:35.839 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:36.074 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.25.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:24:36.075 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-26 2020-07-13T06:24:36.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:36.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:36.078 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:36.124 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:36.125 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:36.382 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.25.6","timestamp":"2020-07-13T06:24:18.598609Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:24:36.382 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-26 2020-07-13T06:24:36.383 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-26 zones={test-zone2, test-zone1} 2020-07-13T06:24:37.638 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:24:37.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:37.652 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:24:37.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:37.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:37.824 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-26 zones={test-zone2, test-zone1} 2020-07-13T06:24:37.825 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-26 zones={test-zone3, test-zone1} 2020-07-13T06:24:38.183 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:24:38.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:38.231 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:24:38.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:38.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:38.309 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-26 zones={test-zone3, test-zone1} 2020-07-13T06:24:38.309 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-26 zones={test-zone3, test-zone2} 2020-07-13T06:24:38.321 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:24:38.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:38.371 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:24:38.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-13T06:24:38.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-13T06:24:38.443 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-26 zones={test-zone3, test-zone2} 2020-07-13T06:24:38.444 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2020-07-13T06:24:38.444 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-27 2020-07-13T06:24:38.479 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-28 2020-07-13T06:24:38.533 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-29 2020-07-13T06:24:38.782 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2020-07-13T06:24:38.783 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-30 2020-07-13T06:24:38.820 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-31 2020-07-13T06:24:38.917 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-32 2020-07-13T06:24:39.174 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2020-07-13T06:24:39.175 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-33 2020-07-13T06:24:39.216 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-34 2020-07-13T06:24:39.303 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-35 2020-07-13T06:24:42.544 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2020-07-13T06:24:42.545 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-36 2020-07-13T06:24:42.931 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-37 2020-07-13T06:24:43.051 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-38 2020-07-13T06:24:43.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:43.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:43.124 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:43.158 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:43.342 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=587bd98b-69d2-49ba-9dbd-2be72177aade 2020-07-13T06:24:43.374 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621483.093117_620.1","last_update":"2020-07-13T06:24:43.093117Z"},{"marker":"1_1594620933.443793_144.1","last_update":"2020-07-13T06:15:33.443793Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:24:43.375 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:24:43.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:43.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:43.378 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:43.420 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:43.421 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:43.650 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621483.093117_620.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:24:43.093117Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:24:43.651 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:24:43.652 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:24:43.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:43.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:43.655 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:43.698 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:43.699 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:43.927 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621483.093117_620.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:24:43.093117Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:24:43.927 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:24:43.928 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:24:43.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:43.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:43.930 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:43.976 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:43.977 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:44.247 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621483.093117_620.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:24:43.093117Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:24:44.248 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:24:46.447 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2020-07-13T06:24:46.448 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-39 2020-07-13T06:24:46.477 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-40 2020-07-13T06:24:46.795 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-41 2020-07-13T06:24:46.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:46.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:46.855 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:46.894 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:46.894 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:47.072 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=587bd98b-69d2-49ba-9dbd-2be72177aade 2020-07-13T06:24:47.109 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621486.829707_634.1","last_update":"2020-07-13T06:24:46.829707Z"},{"marker":"1_1594620933.443793_144.1","last_update":"2020-07-13T06:15:33.443793Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:24:47.109 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:24:47.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:47.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:47.112 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:47.152 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:47.153 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:47.392 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621486.829707_634.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:24:46.829707Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:24:47.393 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:24:47.393 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:24:47.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:47.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:47.395 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:47.435 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:47.436 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:47.668 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621486.829707_634.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:24:46.829707Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:24:47.669 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:24:47.669 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:24:47.670 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:47.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:24:47.671 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:24:47.713 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:47.714 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:47.982 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"587bd98b-69d2-49ba-9dbd-2be72177aade","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621486.829707_634.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:24:46.829707Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:24:47.982 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:24:47.984 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-07-13T06:24:47.984 INFO:teuthology.orchestra.run:waiting for 300 2020-07-13T06:24:47.986 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2020-07-13T06:24:47.980+0000 7f1643fff700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 16309) UID: 0 2020-07-13T06:24:47.986 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2020-07-13T06:24:47.980+0000 7f165fd2c100 -1 shutting down 2020-07-13T06:24:49.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-07-13T06:24:54.087 INFO:tasks.rgw.c1.client.1:Stopped 2020-07-13T06:24:54.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:24:54.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:24:54.090 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:24:54.133 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:54.134 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:54.162 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:24:54.155+0000 7fc644aaf0c0 0 NOTICE: overriding master zone: 5fd641b5-9375-4754-8474-97ebd79eafb6 2020-07-13T06:24:54.163 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:24:54.155+0000 7fc644aaf0c0 0 failed reading obj info from .rgw.root:zone_info.5fd641b5-9375-4754-8474-97ebd79eafb6: (2) No such file or directory 2020-07-13T06:24:54.163 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:24:54.155+0000 7fc644aaf0c0 0 WARNING: could not read zone params for zone id=5fd641b5-9375-4754-8474-97ebd79eafb6 name=test-zone1 2020-07-13T06:24:54.167 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:24:54.163+0000 7fc644aaf0c0 0 failed reading obj info from .rgw.root:zone_info.e66a1e9b-6861-42ff-8eed-7287194f3fe0: (2) No such file or directory 2020-07-13T06:24:54.168 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:24:54.163+0000 7fc644aaf0c0 0 WARNING: could not read zone params for zone id=e66a1e9b-6861-42ff-8eed-7287194f3fe0 name=test-zone3 2020-07-13T06:24:54.180 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"HVCB2URARHK5PDP7","secret_key":"iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668"} 2020-07-13T06:24:54.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:24:54.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:24:54.182 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:24:54.262 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:24:54.263 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:24:54.561 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"c0ad3871-076a-4736-ba2c-6cf14ab7b93e","epoch":1,"predecessor_uuid":"587bd98b-69d2-49ba-9dbd-2be72177aade","sync_status":["1_1594621486.829707_634.1","1_1594620933.443793_144.1","",""],"period_map":{"id":"c0ad3871-076a-4736-ba2c-6cf14ab7b93e","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","zones":[{"id":"5fd641b5-9375-4754-8474-97ebd79eafb6","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","endpoints":["http://smithi014.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","name":"test-zone4","endpoints":["http://smithi014.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5fd641b5-9375-4754-8474-97ebd79eafb6","val":1497020214},{"key":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","val":508986481},{"key":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","val":2882337890},{"key":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","val":617933293}]},"master_zonegroup":"test-zonegroup","master_zone":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","realm_name":"test-realm","realm_epoch":3} 2020-07-13T06:24:54.562 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2020-07-13T06:24:55.202 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-07-13T06:25:54.564 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-42 2020-07-13T06:25:54.644 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-43 2020-07-13T06:25:54.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:25:54.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:25:54.686 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:25:54.722 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:25:54.723 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:25:54.916 INFO:teuthology.orchestra.run.smithi014.stderr:No --period given, using current period=c0ad3871-076a-4736-ba2c-6cf14ab7b93e 2020-07-13T06:25:54.953 INFO:teuthology.orchestra.run.smithi014.stdout:[{"marker":"1_1594621554.679151_423.1","last_update":"2020-07-13T06:25:54.679151Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:25:54.954 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2020-07-13T06:25:54.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:25:54.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:25:54.955 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:25:54.996 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:25:54.996 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:25:55.237 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c0ad3871-076a-4736-ba2c-6cf14ab7b93e","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621554.679151_423.1","next_step_marker":"","total_entries":81,"pos":0,"timestamp":"2020-07-13T06:25:54.679151Z","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":81,"complete":81}} 2020-07-13T06:25:55.238 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2020-07-13T06:25:55.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:25:55.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:25:55.240 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:25:55.316 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:25:55.316 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:25:55.344 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:25:55.339+0000 7f601a8cb0c0 0 NOTICE: overriding master zone: 7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e 2020-07-13T06:25:55.346 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:25:55.339+0000 7f601a8cb0c0 0 failed reading obj info from .rgw.root:zone_info.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e: (2) No such file or directory 2020-07-13T06:25:55.346 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:25:55.339+0000 7f601a8cb0c0 0 WARNING: could not read zone params for zone id=7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e name=test-zone2 2020-07-13T06:25:55.347 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:25:55.343+0000 7f601a8cb0c0 0 failed reading obj info from .rgw.root:zone_info.d4145d87-2321-4625-b4d5-2fd46dfc82ef: (2) No such file or directory 2020-07-13T06:25:55.347 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:25:55.343+0000 7f601a8cb0c0 0 WARNING: could not read zone params for zone id=d4145d87-2321-4625-b4d5-2fd46dfc82ef name=test-zone4 2020-07-13T06:25:55.361 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"5fd641b5-9375-4754-8474-97ebd79eafb6","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"HVCB2URARHK5PDP7","secret_key":"iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668"} 2020-07-13T06:25:55.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:25:55.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:25:55.363 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:25:55.442 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:25:55.442 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:25:55.758 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"9ca21b56-c204-42f4-9360-46419e3ebcc4","epoch":1,"predecessor_uuid":"c0ad3871-076a-4736-ba2c-6cf14ab7b93e","sync_status":["1_1594621554.679151_423.1","","",""],"period_map":{"id":"9ca21b56-c204-42f4-9360-46419e3ebcc4","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","zones":[{"id":"5fd641b5-9375-4754-8474-97ebd79eafb6","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","endpoints":["http://smithi014.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","name":"test-zone4","endpoints":["http://smithi014.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5fd641b5-9375-4754-8474-97ebd79eafb6","val":1497020214},{"key":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","val":508986481},{"key":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","val":2882337890},{"key":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","val":617933293}]},"master_zonegroup":"test-zonegroup","master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","realm_name":"test-realm","realm_epoch":4} 2020-07-13T06:25:55.759 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2020-07-13T06:26:55.762 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-44 2020-07-13T06:26:55.827 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-45 2020-07-13T06:26:55.912 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-07-13T06:26:55.913 INFO:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-07-13T06:26:55.928 INFO:tasks.rgw.c1.client.1:Started 2020-07-13T06:26:55.929 INFO:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2020-07-13T06:26:55.951 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-13T06:26:55.952 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2020-07-13T06:26:55.959 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi017.front.sepia.ceph.com port 8001: Connection refused 2020-07-13T06:26:55.959 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-07-13T06:26:56.961 INFO:teuthology.orchestra.run.smithi017:> curl http://smithi017.front.sepia.ceph.com:8001 2020-07-13T06:26:56.978 INFO:teuthology.orchestra.run.smithi017.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-13T06:26:56.978 INFO:teuthology.orchestra.run.smithi017.stderr: Dload Upload Total Spent Left Speed 2020-07-13T06:26:56.982 INFO:teuthology.orchestra.run.smithi017.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2020-07-13T06:26:56.983 INFO:teuthology.orchestra.run.smithi017.stdout:anonymous 2020-07-13T06:26:56.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:26:56.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:26:56.986 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:26:57.066 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:26:57.067 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:26:57.273 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=9ca21b56-c204-42f4-9360-46419e3ebcc4 2020-07-13T06:26:57.310 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621615.883650_275.1","last_update":"2020-07-13T06:26:55.883650Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:26:57.310 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:26:57.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:26:57.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:26:57.313 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:26:57.363 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:26:57.364 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:26:57.618 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621615.883650_275.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:26:55.883650Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:26:57.619 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:26:57.619 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:26:57.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:26:57.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:26:57.621 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:26:57.662 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:26:57.663 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:26:57.894 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621615.883650_275.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:26:55.883650Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:26:57.894 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:26:57.895 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:26:57.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:26:57.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:26:57.897 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:26:57.937 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:26:57.938 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:26:58.208 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621615.883650_275.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:26:55.883650Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:26:58.208 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:26:58.209 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-39 zones={test-zone2, test-zone1} 2020-07-13T06:26:59.484 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:26:59.494 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:26:59.598 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-39 zones={test-zone2, test-zone1} 2020-07-13T06:26:59.599 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-39 zones={test-zone3, test-zone1} 2020-07-13T06:26:59.616 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:26:59.666 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:26:59.735 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-39 zones={test-zone3, test-zone1} 2020-07-13T06:26:59.736 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-39 zones={test-zone1, test-zone2} 2020-07-13T06:26:59.747 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:26:59.798 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:26:59.864 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-39 zones={test-zone1, test-zone2} 2020-07-13T06:26:59.865 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-39 zones={test-zone3, test-zone2} 2020-07-13T06:26:59.876 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:26:59.926 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:26:59.993 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-39 zones={test-zone3, test-zone2} 2020-07-13T06:26:59.994 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-39 zones={test-zone1, test-zone3} 2020-07-13T06:27:00.011 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:00.070 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:00.135 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-39 zones={test-zone1, test-zone3} 2020-07-13T06:27:00.136 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-39 zones={test-zone2, test-zone3} 2020-07-13T06:27:00.149 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:00.198 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:00.267 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-39 zones={test-zone2, test-zone3} 2020-07-13T06:27:00.267 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-40 zones={test-zone2, test-zone1} 2020-07-13T06:27:00.282 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:00.330 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:00.414 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-40 zones={test-zone2, test-zone1} 2020-07-13T06:27:00.415 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-40 zones={test-zone3, test-zone1} 2020-07-13T06:27:00.429 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:00.478 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:00.552 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-40 zones={test-zone3, test-zone1} 2020-07-13T06:27:00.552 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-40 zones={test-zone1, test-zone2} 2020-07-13T06:27:00.569 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:00.626 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:00.693 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-40 zones={test-zone1, test-zone2} 2020-07-13T06:27:00.694 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-40 zones={test-zone3, test-zone2} 2020-07-13T06:27:00.704 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:00.758 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:00.829 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-40 zones={test-zone3, test-zone2} 2020-07-13T06:27:00.830 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-40 zones={test-zone1, test-zone3} 2020-07-13T06:27:00.843 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:00.891 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:00.959 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-40 zones={test-zone1, test-zone3} 2020-07-13T06:27:00.960 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-40 zones={test-zone2, test-zone3} 2020-07-13T06:27:00.970 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:01.019 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:01.093 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-40 zones={test-zone2, test-zone3} 2020-07-13T06:27:01.094 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-41 zones={test-zone2, test-zone1} 2020-07-13T06:27:01.114 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:01.162 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:01.232 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-41 zones={test-zone2, test-zone1} 2020-07-13T06:27:01.233 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-41 zones={test-zone3, test-zone1} 2020-07-13T06:27:01.243 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:01.295 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:01.364 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-41 zones={test-zone3, test-zone1} 2020-07-13T06:27:01.365 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-41 zones={test-zone1, test-zone2} 2020-07-13T06:27:01.375 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:01.426 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:01.498 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-41 zones={test-zone1, test-zone2} 2020-07-13T06:27:01.499 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-41 zones={test-zone3, test-zone2} 2020-07-13T06:27:01.508 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:01.558 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:01.630 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-41 zones={test-zone3, test-zone2} 2020-07-13T06:27:01.630 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-41 zones={test-zone1, test-zone3} 2020-07-13T06:27:01.643 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:01.703 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:01.771 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-41 zones={test-zone1, test-zone3} 2020-07-13T06:27:01.772 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-41 zones={test-zone2, test-zone3} 2020-07-13T06:27:01.784 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:01.834 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:01.903 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-41 zones={test-zone2, test-zone3} 2020-07-13T06:27:01.903 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-42 zones={test-zone2, test-zone1} 2020-07-13T06:27:01.914 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:01.966 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:02.036 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-42 zones={test-zone2, test-zone1} 2020-07-13T06:27:02.036 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-42 zones={test-zone3, test-zone1} 2020-07-13T06:27:02.046 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:02.098 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:02.166 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-42 zones={test-zone3, test-zone1} 2020-07-13T06:27:02.166 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-42 zones={test-zone1, test-zone2} 2020-07-13T06:27:02.177 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:02.226 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:02.293 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-42 zones={test-zone1, test-zone2} 2020-07-13T06:27:02.293 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-42 zones={test-zone3, test-zone2} 2020-07-13T06:27:02.303 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:02.359 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:02.429 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-42 zones={test-zone3, test-zone2} 2020-07-13T06:27:02.430 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-42 zones={test-zone1, test-zone3} 2020-07-13T06:27:02.443 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:02.490 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:02.561 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-42 zones={test-zone1, test-zone3} 2020-07-13T06:27:02.562 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-42 zones={test-zone2, test-zone3} 2020-07-13T06:27:02.573 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:02.622 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:02.690 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-42 zones={test-zone2, test-zone3} 2020-07-13T06:27:02.691 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-43 zones={test-zone2, test-zone1} 2020-07-13T06:27:02.702 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:02.754 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:02.823 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-43 zones={test-zone2, test-zone1} 2020-07-13T06:27:02.823 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-43 zones={test-zone3, test-zone1} 2020-07-13T06:27:02.833 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:02.882 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:02.950 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-43 zones={test-zone3, test-zone1} 2020-07-13T06:27:02.951 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-43 zones={test-zone1, test-zone2} 2020-07-13T06:27:02.961 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:03.010 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:03.081 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-43 zones={test-zone1, test-zone2} 2020-07-13T06:27:03.082 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-43 zones={test-zone3, test-zone2} 2020-07-13T06:27:03.093 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:03.142 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:03.209 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-43 zones={test-zone3, test-zone2} 2020-07-13T06:27:03.210 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-43 zones={test-zone1, test-zone3} 2020-07-13T06:27:03.223 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:03.274 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:03.338 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-43 zones={test-zone1, test-zone3} 2020-07-13T06:27:03.339 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-43 zones={test-zone2, test-zone3} 2020-07-13T06:27:03.350 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:03.398 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:03.467 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-43 zones={test-zone2, test-zone3} 2020-07-13T06:27:03.468 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-44 zones={test-zone2, test-zone1} 2020-07-13T06:27:03.483 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:03.537 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:03.607 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-44 zones={test-zone2, test-zone1} 2020-07-13T06:27:03.608 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-44 zones={test-zone3, test-zone1} 2020-07-13T06:27:03.616 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:03.663 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:03.730 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-44 zones={test-zone3, test-zone1} 2020-07-13T06:27:03.731 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-44 zones={test-zone1, test-zone2} 2020-07-13T06:27:03.749 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:03.798 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:03.865 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-44 zones={test-zone1, test-zone2} 2020-07-13T06:27:03.867 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-44 zones={test-zone3, test-zone2} 2020-07-13T06:27:03.877 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:03.926 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:03.996 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-44 zones={test-zone3, test-zone2} 2020-07-13T06:27:03.997 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-44 zones={test-zone1, test-zone3} 2020-07-13T06:27:04.009 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:04.058 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:04.122 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-44 zones={test-zone1, test-zone3} 2020-07-13T06:27:04.123 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-44 zones={test-zone2, test-zone3} 2020-07-13T06:27:04.134 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:04.182 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:04.252 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-44 zones={test-zone2, test-zone3} 2020-07-13T06:27:04.253 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-45 zones={test-zone2, test-zone1} 2020-07-13T06:27:04.264 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:04.314 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:04.391 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-45 zones={test-zone2, test-zone1} 2020-07-13T06:27:04.392 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-45 zones={test-zone3, test-zone1} 2020-07-13T06:27:04.406 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:04.454 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:04.521 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-45 zones={test-zone3, test-zone1} 2020-07-13T06:27:04.522 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-45 zones={test-zone1, test-zone2} 2020-07-13T06:27:04.533 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:04.586 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:04.650 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-45 zones={test-zone1, test-zone2} 2020-07-13T06:27:04.650 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-45 zones={test-zone3, test-zone2} 2020-07-13T06:27:04.663 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:04.714 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:04.782 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-45 zones={test-zone3, test-zone2} 2020-07-13T06:27:04.783 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-45 zones={test-zone1, test-zone3} 2020-07-13T06:27:04.800 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:04.850 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:04.928 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-45 zones={test-zone1, test-zone3} 2020-07-13T06:27:04.929 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-45 zones={test-zone2, test-zone3} 2020-07-13T06:27:04.942 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:27:04.994 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:27:05.070 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-45 zones={test-zone2, test-zone3} 2020-07-13T06:27:05.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '587bd98b-69d2-49ba-9dbd-2be72177aade', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:05.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '587bd98b-69d2-49ba-9dbd-2be72177aade', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:05.072 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 587bd98b-69d2-49ba-9dbd-2be72177aade --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:05.108 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:05.109 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:05.407 INFO:teuthology.orchestra.run.smithi017.stdout:[{"id":"1_1594620792.560320_57.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-07-13T06:13:12.560320432Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w5Jia4Y6RKNUd9E9jGppGgY","ver":1},"status":{"status":"write"}}},{"id":"1_1594620792.569571_63.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-07-13T06:13:12.569571770Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w5Jia4Y6RKNUd9E9jGppGgY","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.353592_71.1","section":"bucket.instance","name":"cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19","timestamp":"2020-07-13T06:15:35.353592485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5YWV65SHwhZo1_hTFsOthdm","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.357969_84.1","section":"bucket.instance","name":"cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19","timestamp":"2020-07-13T06:15:35.357969060Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5YWV65SHwhZo1_hTFsOthdm","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.360453_85.1","section":"bucket","name":"cngrzz-10","timestamp":"2020-07-13T06:15:35.360453117Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lyi7_9F-fQavvS6ttQb4QqQ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.365063_86.1","section":"bucket","name":"cngrzz-10","timestamp":"2020-07-13T06:15:35.365063336Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lyi7_9F-fQavvS6ttQb4QqQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.387630_87.1","section":"bucket.instance","name":"cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20","timestamp":"2020-07-13T06:15:35.387630766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XJOAFF_eewRAyz7Ek55C2eZ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.393964_88.1","section":"bucket.instance","name":"cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20","timestamp":"2020-07-13T06:15:35.393964283Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XJOAFF_eewRAyz7Ek55C2eZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.396127_89.1","section":"bucket","name":"cngrzz-11","timestamp":"2020-07-13T06:15:35.396127319Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uYUbUU7eUol0NqPi_EWhoBS","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.401731_90.1","section":"bucket","name":"cngrzz-11","timestamp":"2020-07-13T06:15:35.401731762Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uYUbUU7eUol0NqPi_EWhoBS","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.439458_91.1","section":"bucket.instance","name":"cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21","timestamp":"2020-07-13T06:15:35.439458898Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y5EI8u7QgEs_npG_hSdZdhk","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.447280_92.1","section":"bucket.instance","name":"cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21","timestamp":"2020-07-13T06:15:35.447280663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y5EI8u7QgEs_npG_hSdZdhk","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.451453_93.1","section":"bucket","name":"cngrzz-12","timestamp":"2020-07-13T06:15:35.451453816Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1xVfXDesczYGyPwaO94C87T","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.457356_94.1","section":"bucket","name":"cngrzz-12","timestamp":"2020-07-13T06:15:35.457356063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1xVfXDesczYGyPwaO94C87T","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.556299_103.1","section":"bucket.instance","name":"cngrzz-13:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.22","timestamp":"2020-07-13T06:15:53.556299812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqQn6_rjhfsJuMkDSydnu0U","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.563929_114.1","section":"bucket.instance","name":"cngrzz-13:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.22","timestamp":"2020-07-13T06:15:53.563929704Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqQn6_rjhfsJuMkDSydnu0U","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.567882_115.1","section":"bucket","name":"cngrzz-13","timestamp":"2020-07-13T06:15:53.567882599Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_raNSztBe-9dO0DB5H_S-No6","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.575383_116.1","section":"bucket","name":"cngrzz-13","timestamp":"2020-07-13T06:15:53.575383568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_raNSztBe-9dO0DB5H_S-No6","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.610132_118.1","section":"bucket.instance","name":"cngrzz-14:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.23","timestamp":"2020-07-13T06:15:53.610132033Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uUpk1HTypA7roPDy4mLwqnM","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.614384_119.1","section":"bucket.instance","name":"cngrzz-14:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.23","timestamp":"2020-07-13T06:15:53.614384828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uUpk1HTypA7roPDy4mLwqnM","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.616654_120.1","section":"bucket","name":"cngrzz-14","timestamp":"2020-07-13T06:15:53.616654760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9zhZRNYkovqDEm3EDgLOk8P","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.622281_121.1","section":"bucket","name":"cngrzz-14","timestamp":"2020-07-13T06:15:53.622281110Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9zhZRNYkovqDEm3EDgLOk8P","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.669597_122.1","section":"bucket.instance","name":"cngrzz-15:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.24","timestamp":"2020-07-13T06:15:53.669597829Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_80eR23_oL7sRegm5pYbqCb3","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.675879_123.1","section":"bucket.instance","name":"cngrzz-15:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.24","timestamp":"2020-07-13T06:15:53.675879396Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_80eR23_oL7sRegm5pYbqCb3","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.677770_124.1","section":"bucket","name":"cngrzz-15","timestamp":"2020-07-13T06:15:53.677770571Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0h_SR2fNa6Li7p9B9ibEN6V","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.682997_125.1","section":"bucket","name":"cngrzz-15","timestamp":"2020-07-13T06:15:53.682997873Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0h_SR2fNa6Li7p9B9ibEN6V","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.903801_180.1","section":"bucket.instance","name":"cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25","timestamp":"2020-07-13T06:16:35.903801109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.908369_187.1","section":"bucket.instance","name":"cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25","timestamp":"2020-07-13T06:16:35.908369870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.910031_188.1","section":"bucket","name":"cngrzz-16","timestamp":"2020-07-13T06:16:35.910031524Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O60kezSGyfXCzbn8fOxwPfi","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.913589_189.1","section":"bucket","name":"cngrzz-16","timestamp":"2020-07-13T06:16:35.913589491Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O60kezSGyfXCzbn8fOxwPfi","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.938308_190.1","section":"bucket.instance","name":"cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26","timestamp":"2020-07-13T06:16:35.938308264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.942759_191.1","section":"bucket.instance","name":"cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26","timestamp":"2020-07-13T06:16:35.942759365Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.944786_192.1","section":"bucket","name":"cngrzz-17","timestamp":"2020-07-13T06:16:35.944786921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uHYXM5KyUTxZhRfiZctWD4n","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.948764_193.1","section":"bucket","name":"cngrzz-17","timestamp":"2020-07-13T06:16:35.948764599Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uHYXM5KyUTxZhRfiZctWD4n","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.990105_194.1","section":"bucket.instance","name":"cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27","timestamp":"2020-07-13T06:16:35.990105813Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.994448_195.1","section":"bucket.instance","name":"cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27","timestamp":"2020-07-13T06:16:35.994448116Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.996481_196.1","section":"bucket","name":"cngrzz-18","timestamp":"2020-07-13T06:16:35.996481980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z5ZtESqqP15O0oomxjuHPqh","ver":1},"status":{"status":"write"}}},{"id":"1_1594620996.000544_197.1","section":"bucket","name":"cngrzz-18","timestamp":"2020-07-13T06:16:36.000544311Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z5ZtESqqP15O0oomxjuHPqh","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620996.023617_198.1","section":"bucket.instance","name":"cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25","timestamp":"2020-07-13T06:16:36.023617389Z","data":{"read_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":1},"write_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":2},"status":{"status":"write"}}},{"id":"1_1594620996.035328_199.1","section":"bucket.instance","name":"cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25","timestamp":"2020-07-13T06:16:36.035328199Z","data":{"read_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":1},"write_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":2},"status":{"status":"complete"}}},{"id":"1_1594620996.042162_200.1","section":"bucket.instance","name":"cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26","timestamp":"2020-07-13T06:16:36.042162134Z","data":{"read_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":1},"write_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":2},"status":{"status":"write"}}},{"id":"1_1594620996.046721_201.1","section":"bucket.instance","name":"cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26","timestamp":"2020-07-13T06:16:36.046721459Z","data":{"read_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":1},"write_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":2},"status":{"status":"complete"}}},{"id":"1_1594620996.060062_202.1","section":"bucket.instance","name":"cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27","timestamp":"2020-07-13T06:16:36.060062705Z","data":{"read_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":1},"write_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":2},"status":{"status":"write"}}},{"id":"1_1594620996.066494_203.1","section":"bucket.instance","name":"cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27","timestamp":"2020-07-13T06:16:36.066494340Z","data":{"read_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":1},"write_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621170.096466_265.1","section":"bucket.instance","name":"cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28","timestamp":"2020-07-13T06:19:30.096466874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":1},"status":{"status":"write"}}},{"id":"1_1594621170.101095_286.1","section":"bucket.instance","name":"cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28","timestamp":"2020-07-13T06:19:30.101095792Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621170.103270_287.1","section":"bucket","name":"cngrzz-19","timestamp":"2020-07-13T06:19:30.103270825Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wQI4zfCNmpgzGLmtq7JYIcF","ver":1},"status":{"status":"write"}}},{"id":"1_1594621170.107053_288.1","section":"bucket","name":"cngrzz-19","timestamp":"2020-07-13T06:19:30.107053912Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wQI4zfCNmpgzGLmtq7JYIcF","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621170.117756_289.1","section":"bucket.instance","name":"cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28","timestamp":"2020-07-13T06:19:30.117756676Z","data":{"read_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":1},"write_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":2},"status":{"status":"write"}}},{"id":"1_1594621170.123767_290.1","section":"bucket.instance","name":"cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28","timestamp":"2020-07-13T06:19:30.123767219Z","data":{"read_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":1},"write_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621322.052070_335.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:22:02.052070538Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":1},"status":{"status":"write"}}},{"id":"1_1594621322.057256_350.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:22:02.057256567Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621322.059913_351.1","section":"bucket","name":"cngrzz-20","timestamp":"2020-07-13T06:22:02.059913372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SiUnqH4drkKjEAXU4lM6KVi","ver":1},"status":{"status":"write"}}},{"id":"1_1594621322.065336_352.1","section":"bucket","name":"cngrzz-20","timestamp":"2020-07-13T06:22:02.065336451Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SiUnqH4drkKjEAXU4lM6KVi","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621329.476722_365.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:22:09.476722756Z","data":{"read_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":1},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":2},"status":{"status":"write"}}},{"id":"1_1594621329.487977_368.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:22:09.487977095Z","data":{"read_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":1},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621390.911893_402.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:23:10.911893325Z","data":{"read_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":2},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":3},"status":{"status":"write"}}},{"id":"1_1594621390.921870_405.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:23:10.921870742Z","data":{"read_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":2},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":3},"status":{"status":"complete"}}},{"id":"1_1594621433.317645_430.1","section":"bucket.instance","name":"cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30","timestamp":"2020-07-13T06:23:53.317645856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":1},"status":{"status":"write"}}},{"id":"1_1594621433.322934_433.1","section":"bucket.instance","name":"cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30","timestamp":"2020-07-13T06:23:53.322934869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621433.324814_434.1","section":"bucket","name":"cngrzz-21","timestamp":"2020-07-13T06:23:53.324814398Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vqw1RgtzigBqWBzMKd9qNek","ver":1},"status":{"status":"write"}}},{"id":"1_1594621433.328408_435.1","section":"bucket","name":"cngrzz-21","timestamp":"2020-07-13T06:23:53.328408882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vqw1RgtzigBqWBzMKd9qNek","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621433.386628_436.1","section":"bucket.instance","name":"cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31","timestamp":"2020-07-13T06:23:53.386628999Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":1},"status":{"status":"write"}}},{"id":"1_1594621433.390873_437.1","section":"bucket.instance","name":"cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31","timestamp":"2020-07-13T06:23:53.390873686Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621433.393237_438.1","section":"bucket","name":"cngrzz-22","timestamp":"2020-07-13T06:23:53.393237663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7bwonnuvmzNIIPT8mR6P0bA","ver":1},"status":{"status":"write"}}},{"id":"1_1594621433.397692_439.1","section":"bucket","name":"cngrzz-22","timestamp":"2020-07-13T06:23:53.397692663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7bwonnuvmzNIIPT8mR6P0bA","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621434.232311_440.1","section":"bucket.instance","name":"cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32","timestamp":"2020-07-13T06:23:54.232311850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":1},"status":{"status":"write"}}},{"id":"1_1594621434.236794_441.1","section":"bucket.instance","name":"cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32","timestamp":"2020-07-13T06:23:54.236794211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621434.238800_442.1","section":"bucket","name":"cngrzz-23","timestamp":"2020-07-13T06:23:54.238800428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SvlhobBY9Slsbwne1CTEC-L","ver":1},"status":{"status":"write"}}},{"id":"1_1594621434.242788_443.1","section":"bucket","name":"cngrzz-23","timestamp":"2020-07-13T06:23:54.242788736Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SvlhobBY9Slsbwne1CTEC-L","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621434.264342_444.1","section":"bucket.instance","name":"cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30","timestamp":"2020-07-13T06:23:54.264342494Z","data":{"read_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":1},"write_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":2},"status":{"status":"write"}}},{"id":"1_1594621434.270547_445.1","section":"bucket.instance","name":"cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30","timestamp":"2020-07-13T06:23:54.270547881Z","data":{"read_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":1},"write_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621434.275368_446.1","section":"bucket.instance","name":"cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31","timestamp":"2020-07-13T06:23:54.275368951Z","data":{"read_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":1},"write_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":2},"status":{"status":"write"}}},{"id":"1_1594621434.282613_447.1","section":"bucket.instance","name":"cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31","timestamp":"2020-07-13T06:23:54.282613618Z","data":{"read_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":1},"write_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621434.294881_448.1","section":"bucket.instance","name":"cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32","timestamp":"2020-07-13T06:23:54.294881064Z","data":{"read_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":1},"write_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":2},"status":{"status":"write"}}},{"id":"1_1594621434.303397_449.1","section":"bucket.instance","name":"cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32","timestamp":"2020-07-13T06:23:54.303397214Z","data":{"read_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":1},"write_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621455.307113_481.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.307113064Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.311789_492.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.311789146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.314390_493.1","section":"bucket","name":"cngrzz-24","timestamp":"2020-07-13T06:24:15.314390695Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_syYOTmiVtbmFjOCG8p3NuzV","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.318379_494.1","section":"bucket","name":"cngrzz-24","timestamp":"2020-07-13T06:24:15.318379915Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_syYOTmiVtbmFjOCG8p3NuzV","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.344614_495.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.344614344Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.349148_496.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.349148544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.351560_497.1","section":"bucket","name":"cngrzz-25","timestamp":"2020-07-13T06:24:15.351560923Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L6eQ90-G-twMkpkPL-m5q_J","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.355886_498.1","section":"bucket","name":"cngrzz-25","timestamp":"2020-07-13T06:24:15.355886460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L6eQ90-G-twMkpkPL-m5q_J","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.405945_499.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.405945564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.410863_500.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.410863468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.412859_501.1","section":"bucket","name":"cngrzz-26","timestamp":"2020-07-13T06:24:15.412859756Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wVuwzieS4x5WATempXhxurc","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.417215_502.1","section":"bucket","name":"cngrzz-26","timestamp":"2020-07-13T06:24:15.417215461Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wVuwzieS4x5WATempXhxurc","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.441624_503.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.441624712Z","data":{"read_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":1},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":2},"status":{"status":"write"}}},{"id":"1_1594621455.464024_504.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.464024694Z","data":{"read_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":1},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621455.470869_505.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.470869735Z","data":{"read_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":2},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":3},"status":{"status":"write"}}},{"id":"1_1594621455.475511_506.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.475511044Z","data":{"read_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":2},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":3},"status":{"status":"complete"}}},{"id":"1_1594621455.479588_507.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.479588113Z","data":{"read_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":1},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":2},"status":{"status":"write"}}},{"id":"1_1594621455.484136_508.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.484136677Z","data":{"read_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":1},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621455.494806_509.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.494806032Z","data":{"read_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":2},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":3},"status":{"status":"write"}}},{"id":"1_1594621455.505325_510.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.505325897Z","data":{"read_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":2},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":3},"status":{"status":"complete"}}},{"id":"1_1594621455.518131_511.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.518131893Z","data":{"read_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":1},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":2},"status":{"status":"write"}}},{"id":"1_1594621455.527513_512.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.527513994Z","data":{"read_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":1},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621455.542157_513.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.542157223Z","data":{"read_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":2},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":3},"status":{"status":"write"}}},{"id":"1_1594621455.551984_514.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.551984028Z","data":{"read_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":2},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":3},"status":{"status":"complete"}}},{"id":"1_1594621478.463758_549.1","section":"bucket.instance","name":"cngrzz-27:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.36","timestamp":"2020-07-13T06:24:38.463758191Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.468096_556.1","section":"bucket.instance","name":"cngrzz-27:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.36","timestamp":"2020-07-13T06:24:38.468096388Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.470142_557.1","section":"bucket","name":"cngrzz-27","timestamp":"2020-07-13T06:24:38.470142675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xKmuAPoskvLbXPU03OlC73s","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.473856_558.1","section":"bucket","name":"cngrzz-27","timestamp":"2020-07-13T06:24:38.473856655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xKmuAPoskvLbXPU03OlC73s","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.497896_559.1","section":"bucket.instance","name":"cngrzz-28:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.37","timestamp":"2020-07-13T06:24:38.497896309Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.502362_560.1","section":"bucket.instance","name":"cngrzz-28:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.37","timestamp":"2020-07-13T06:24:38.502362092Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.504706_561.1","section":"bucket","name":"cngrzz-28","timestamp":"2020-07-13T06:24:38.504706153Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m0vuxwdxcV4K0zwl3eB9Wa9","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.509299_562.1","section":"bucket","name":"cngrzz-28","timestamp":"2020-07-13T06:24:38.509299001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m0vuxwdxcV4K0zwl3eB9Wa9","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.554287_563.1","section":"bucket.instance","name":"cngrzz-29:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.38","timestamp":"2020-07-13T06:24:38.554287987Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.561718_564.1","section":"bucket.instance","name":"cngrzz-29:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.38","timestamp":"2020-07-13T06:24:38.561718878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.564680_565.1","section":"bucket","name":"cngrzz-29","timestamp":"2020-07-13T06:24:38.564680554Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2TuYKigZrzuv-dMOyxValfc","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.569842_566.1","section":"bucket","name":"cngrzz-29","timestamp":"2020-07-13T06:24:38.569842326Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2TuYKigZrzuv-dMOyxValfc","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.595037_567.1","section":"bucket.instance","name":"cngrzz-27:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.36","timestamp":"2020-07-13T06:24:38.595037740Z","data":{"read_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":1},"write_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":2},"status":{"status":"write"}}},{"id":"1_1594621478.600762_568.1","section":"bucket.instance","name":"cngrzz-27:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.36","timestamp":"2020-07-13T06:24:38.600762996Z","data":{"read_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":1},"write_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621478.647960_569.1","section":"bucket.instance","name":"cngrzz-28:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.37","timestamp":"2020-07-13T06:24:38.647960510Z","data":{"read_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":1},"write_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":2},"status":{"status":"write"}}},{"id":"1_1594621478.658940_570.1","section":"bucket.instance","name":"cngrzz-28:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.37","timestamp":"2020-07-13T06:24:38.658940233Z","data":{"read_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":1},"write_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621478.723174_571.1","section":"bucket.instance","name":"cngrzz-29:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.38","timestamp":"2020-07-13T06:24:38.723174021Z","data":{"read_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":1},"write_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":2},"status":{"status":"write"}}},{"id":"1_1594621478.729364_572.1","section":"bucket.instance","name":"cngrzz-29:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.38","timestamp":"2020-07-13T06:24:38.729364605Z","data":{"read_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":1},"write_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621478.803680_573.1","section":"bucket.instance","name":"cngrzz-30:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.39","timestamp":"2020-07-13T06:24:38.803680169Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.808550_574.1","section":"bucket.instance","name":"cngrzz-30:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.39","timestamp":"2020-07-13T06:24:38.808550868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.810788_575.1","section":"bucket","name":"cngrzz-30","timestamp":"2020-07-13T06:24:38.810788109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tcbDPAiLUbg1Qosf4aVJZaR","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.814964_576.1","section":"bucket","name":"cngrzz-30","timestamp":"2020-07-13T06:24:38.814964209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tcbDPAiLUbg1Qosf4aVJZaR","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.842441_577.1","section":"bucket.instance","name":"cngrzz-31:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.40","timestamp":"2020-07-13T06:24:38.842441698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.846740_578.1","section":"bucket.instance","name":"cngrzz-31:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.40","timestamp":"2020-07-13T06:24:38.846740063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.849242_579.1","section":"bucket","name":"cngrzz-31","timestamp":"2020-07-13T06:24:38.849242603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PcNMXYzKJiqzUL9v_g9UUEW","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.853963_580.1","section":"bucket","name":"cngrzz-31","timestamp":"2020-07-13T06:24:38.853963184Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PcNMXYzKJiqzUL9v_g9UUEW","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.938346_581.1","section":"bucket.instance","name":"cngrzz-32:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.41","timestamp":"2020-07-13T06:24:38.938346254Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.942593_582.1","section":"bucket.instance","name":"cngrzz-32:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.41","timestamp":"2020-07-13T06:24:38.942593554Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.944644_583.1","section":"bucket","name":"cngrzz-32","timestamp":"2020-07-13T06:24:38.944644272Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cgRpl2sxiftnVUa2OO_gEgo","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.948537_584.1","section":"bucket","name":"cngrzz-32","timestamp":"2020-07-13T06:24:38.948537812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cgRpl2sxiftnVUa2OO_gEgo","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.974066_585.1","section":"bucket.instance","name":"cngrzz-30:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.39","timestamp":"2020-07-13T06:24:38.974066414Z","data":{"read_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":1},"write_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":2},"status":{"status":"write"}}},{"id":"1_1594621478.980780_586.1","section":"bucket.instance","name":"cngrzz-30:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.39","timestamp":"2020-07-13T06:24:38.980780610Z","data":{"read_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":1},"write_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621479.033952_587.1","section":"bucket.instance","name":"cngrzz-31:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.40","timestamp":"2020-07-13T06:24:39.033952236Z","data":{"read_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":1},"write_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":2},"status":{"status":"write"}}},{"id":"1_1594621479.045331_588.1","section":"bucket.instance","name":"cngrzz-31:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.40","timestamp":"2020-07-13T06:24:39.045331654Z","data":{"read_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":1},"write_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621479.109134_589.1","section":"bucket.instance","name":"cngrzz-32:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.41","timestamp":"2020-07-13T06:24:39.109134981Z","data":{"read_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":1},"write_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":2},"status":{"status":"write"}}},{"id":"1_1594621479.118646_590.1","section":"bucket.instance","name":"cngrzz-32:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.41","timestamp":"2020-07-13T06:24:39.118646708Z","data":{"read_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":1},"write_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621479.194094_591.1","section":"bucket.instance","name":"cngrzz-33:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.42","timestamp":"2020-07-13T06:24:39.194094933Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.200722_592.1","section":"bucket.instance","name":"cngrzz-33:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.42","timestamp":"2020-07-13T06:24:39.200722310Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.203027_593.1","section":"bucket","name":"cngrzz-33","timestamp":"2020-07-13T06:24:39.203027753Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PuawPXsI3tXqBy_MXXvr0Cy","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.209920_594.1","section":"bucket","name":"cngrzz-33","timestamp":"2020-07-13T06:24:39.209920207Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PuawPXsI3tXqBy_MXXvr0Cy","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.241265_595.1","section":"bucket.instance","name":"cngrzz-34:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.43","timestamp":"2020-07-13T06:24:39.241265494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.246173_596.1","section":"bucket.instance","name":"cngrzz-34:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.43","timestamp":"2020-07-13T06:24:39.246173690Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.248646_597.1","section":"bucket","name":"cngrzz-34","timestamp":"2020-07-13T06:24:39.248646104Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SZllY3UvM6YWJqOBtYiiacb","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.253617_598.1","section":"bucket","name":"cngrzz-34","timestamp":"2020-07-13T06:24:39.253617145Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SZllY3UvM6YWJqOBtYiiacb","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.323567_599.1","section":"bucket.instance","name":"cngrzz-35:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.44","timestamp":"2020-07-13T06:24:39.323567314Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.328276_600.1","section":"bucket.instance","name":"cngrzz-35:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.44","timestamp":"2020-07-13T06:24:39.328276870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.330806_601.1","section":"bucket","name":"cngrzz-35","timestamp":"2020-07-13T06:24:39.330806173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v-R90_jF6GVW73YIo8cHfq0","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.336092_602.1","section":"bucket","name":"cngrzz-35","timestamp":"2020-07-13T06:24:39.336092549Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v-R90_jF6GVW73YIo8cHfq0","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.359801_603.1","section":"bucket.instance","name":"cngrzz-33:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.42","timestamp":"2020-07-13T06:24:39.359801153Z","data":{"read_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":1},"write_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":2},"status":{"status":"write"}}},{"id":"1_1594621479.369370_604.1","section":"bucket.instance","name":"cngrzz-33:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.42","timestamp":"2020-07-13T06:24:39.369370295Z","data":{"read_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":1},"write_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621480.420657_605.1","section":"bucket.instance","name":"cngrzz-34:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.43","timestamp":"2020-07-13T06:24:40.420657898Z","data":{"read_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":1},"write_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":2},"status":{"status":"write"}}},{"id":"1_1594621480.432601_606.1","section":"bucket.instance","name":"cngrzz-34:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.43","timestamp":"2020-07-13T06:24:40.432601102Z","data":{"read_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":1},"write_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621482.474186_607.1","section":"bucket.instance","name":"cngrzz-35:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.44","timestamp":"2020-07-13T06:24:42.474186915Z","data":{"read_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":1},"write_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":2},"status":{"status":"write"}}},{"id":"1_1594621482.484822_608.1","section":"bucket.instance","name":"cngrzz-35:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.44","timestamp":"2020-07-13T06:24:42.484822401Z","data":{"read_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":1},"write_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621482.913401_609.1","section":"bucket.instance","name":"cngrzz-36:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.45","timestamp":"2020-07-13T06:24:42.913401257Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BHYIPGqLECVNEi2cZ-MYjOz","ver":1},"status":{"status":"write"}}},{"id":"1_1594621482.917800_610.1","section":"bucket.instance","name":"cngrzz-36:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.45","timestamp":"2020-07-13T06:24:42.917800056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BHYIPGqLECVNEi2cZ-MYjOz","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621482.920043_611.1","section":"bucket","name":"cngrzz-36","timestamp":"2020-07-13T06:24:42.920043693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lDf1FKkxnRubfsQ9M1--dgD","ver":1},"status":{"status":"write"}}},{"id":"1_1594621482.924506_612.1","section":"bucket","name":"cngrzz-36","timestamp":"2020-07-13T06:24:42.924506470Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lDf1FKkxnRubfsQ9M1--dgD","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621483.015114_613.1","section":"bucket.instance","name":"cngrzz-37:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.46","timestamp":"2020-07-13T06:24:43.015114675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oP_SUc8PxL6c4XMdnGrcIDd","ver":1},"status":{"status":"write"}}},{"id":"1_1594621483.020538_614.1","section":"bucket.instance","name":"cngrzz-37:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.46","timestamp":"2020-07-13T06:24:43.020538172Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oP_SUc8PxL6c4XMdnGrcIDd","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621483.022545_615.1","section":"bucket","name":"cngrzz-37","timestamp":"2020-07-13T06:24:43.022545950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5gFmRJU_Lnfca_c3ktRTso7","ver":1},"status":{"status":"write"}}},{"id":"1_1594621483.026371_616.1","section":"bucket","name":"cngrzz-37","timestamp":"2020-07-13T06:24:43.026371722Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5gFmRJU_Lnfca_c3ktRTso7","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621483.081477_617.1","section":"bucket.instance","name":"cngrzz-38:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.47","timestamp":"2020-07-13T06:24:43.081477070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nVm3w2ql0Kn2oqc2UsEJH8a","ver":1},"status":{"status":"write"}}},{"id":"1_1594621483.087242_618.1","section":"bucket.instance","name":"cngrzz-38:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.47","timestamp":"2020-07-13T06:24:43.087242473Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nVm3w2ql0Kn2oqc2UsEJH8a","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621483.089407_619.1","section":"bucket","name":"cngrzz-38","timestamp":"2020-07-13T06:24:43.089407632Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3eyVOL0MMcQBz9oFqM87zxz","ver":1},"status":{"status":"write"}}},{"id":"1_1594621483.093117_620.1","section":"bucket","name":"cngrzz-38","timestamp":"2020-07-13T06:24:43.093117980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3eyVOL0MMcQBz9oFqM87zxz","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.461368_621.1","section":"bucket.instance","name":"cngrzz-39:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.48","timestamp":"2020-07-13T06:24:46.461368112Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uMnm_CbH26xOW2j2VokpY4s","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.466851_624.1","section":"bucket.instance","name":"cngrzz-39:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.48","timestamp":"2020-07-13T06:24:46.466851306Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uMnm_CbH26xOW2j2VokpY4s","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.469013_625.1","section":"bucket","name":"cngrzz-39","timestamp":"2020-07-13T06:24:46.469013868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eqEaW0o9d4QYozEF28u7c49","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.472599_626.1","section":"bucket","name":"cngrzz-39","timestamp":"2020-07-13T06:24:46.472599369Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eqEaW0o9d4QYozEF28u7c49","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.759727_627.1","section":"bucket.instance","name":"cngrzz-40:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.49","timestamp":"2020-07-13T06:24:46.759727960Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OP3txqbvev88w0dP9vK-dO5","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.765631_628.1","section":"bucket.instance","name":"cngrzz-40:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.49","timestamp":"2020-07-13T06:24:46.765631411Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OP3txqbvev88w0dP9vK-dO5","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.767996_629.1","section":"bucket","name":"cngrzz-40","timestamp":"2020-07-13T06:24:46.767996150Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M3LKRjOIblMfECEncxsHzyu","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.772433_630.1","section":"bucket","name":"cngrzz-40","timestamp":"2020-07-13T06:24:46.772433975Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M3LKRjOIblMfECEncxsHzyu","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.817605_631.1","section":"bucket.instance","name":"cngrzz-41:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.50","timestamp":"2020-07-13T06:24:46.817605518Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kHTWaobdOcV3WvsIXS-4JDw","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.823059_632.1","section":"bucket.instance","name":"cngrzz-41:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.50","timestamp":"2020-07-13T06:24:46.823059818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kHTWaobdOcV3WvsIXS-4JDw","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.825455_633.1","section":"bucket","name":"cngrzz-41","timestamp":"2020-07-13T06:24:46.825455311Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_enzpzPLLpAzWWe8NZI6ZjKh","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.829707_634.1","section":"bucket","name":"cngrzz-41","timestamp":"2020-07-13T06:24:46.829707023Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_enzpzPLLpAzWWe8NZI6ZjKh","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620914.565843_67.1","section":"bucket.instance","name":"cngrzz-1:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.1","timestamp":"2020-07-13T06:15:14.565843290Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NrX7NUBlKlctPxLgZQhVZeJ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620914.569624_68.1","section":"bucket.instance","name":"cngrzz-1:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.1","timestamp":"2020-07-13T06:15:14.569624562Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NrX7NUBlKlctPxLgZQhVZeJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620914.571621_69.1","section":"bucket","name":"cngrzz-1","timestamp":"2020-07-13T06:15:14.571621151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B_xEGfjqp5GJf-kckzrymxy","ver":1},"status":{"status":"write"}}},{"id":"1_1594620914.576799_70.1","section":"bucket","name":"cngrzz-1","timestamp":"2020-07-13T06:15:14.576799306Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B_xEGfjqp5GJf-kckzrymxy","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620914.604902_71.1","section":"bucket.instance","name":"cngrzz-2:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.2","timestamp":"2020-07-13T06:15:14.604902219Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PvZsoXHBEggBhfEef7o5U0T","ver":1},"status":{"status":"write"}}},{"id":"1_1594620914.609491_72.1","section":"bucket.instance","name":"cngrzz-2:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.2","timestamp":"2020-07-13T06:15:14.609491570Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PvZsoXHBEggBhfEef7o5U0T","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620914.612054_73.1","section":"bucket","name":"cngrzz-2","timestamp":"2020-07-13T06:15:14.612054698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6IT0MMiYAzd8ADA6mJQGl2V","ver":1},"status":{"status":"write"}}},{"id":"1_1594620914.620627_74.1","section":"bucket","name":"cngrzz-2","timestamp":"2020-07-13T06:15:14.620627832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6IT0MMiYAzd8ADA6mJQGl2V","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620918.616085_75.1","section":"bucket.instance","name":"cngrzz-3:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.3","timestamp":"2020-07-13T06:15:18.616085899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jNBNF8gIfCQ0VtU1Nd6VsXo","ver":1},"status":{"status":"write"}}},{"id":"1_1594620918.620674_76.1","section":"bucket.instance","name":"cngrzz-3:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.3","timestamp":"2020-07-13T06:15:18.620674736Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jNBNF8gIfCQ0VtU1Nd6VsXo","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620918.625187_77.1","section":"bucket","name":"cngrzz-3","timestamp":"2020-07-13T06:15:18.625187717Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6UxFHip67VoBBxh6FLU5ArZ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620918.629897_78.1","section":"bucket","name":"cngrzz-3","timestamp":"2020-07-13T06:15:18.629897499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6UxFHip67VoBBxh6FLU5ArZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.868689_79.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.4","timestamp":"2020-07-13T06:15:21.868689028Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5eRMS50W9omalXkI-eKEkZH","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.873048_80.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.4","timestamp":"2020-07-13T06:15:21.873048233Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5eRMS50W9omalXkI-eKEkZH","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.874813_81.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:21.874813555Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JboyausbemE9wke0D-OFRJL","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.881925_82.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:21.881925098Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JboyausbemE9wke0D-OFRJL","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.904963_83.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.5","timestamp":"2020-07-13T06:15:21.904963978Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_orhbCdPThQDntboRt2h0Sl9","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.910649_84.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.5","timestamp":"2020-07-13T06:15:21.910649089Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_orhbCdPThQDntboRt2h0Sl9","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.913196_85.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:21.913196616Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W9AKCYVnn-UeTSsMsys7MT_","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.920774_86.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:21.920774888Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W9AKCYVnn-UeTSsMsys7MT_","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.970374_87.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.6","timestamp":"2020-07-13T06:15:21.970374493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v6ZB9vPFQIwn4oxtloegK6s","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.976019_88.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.6","timestamp":"2020-07-13T06:15:21.976019891Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v6ZB9vPFQIwn4oxtloegK6s","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.980716_89.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:21.980716496Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IjM-H8ZOw8IgEVRDW0EGVzm","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.991312_90.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:21.991312189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IjM-H8ZOw8IgEVRDW0EGVzm","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.389888_91.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.7","timestamp":"2020-07-13T06:15:25.389888371Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RWdhBH265JDcuhXxt8yJvwV","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.394176_92.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.7","timestamp":"2020-07-13T06:15:25.394176539Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RWdhBH265JDcuhXxt8yJvwV","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.396386_93.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:25.396386821Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t0iFgy3ChisA7Hu7SEB54i8","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.400663_94.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:25.400663561Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t0iFgy3ChisA7Hu7SEB54i8","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.431105_95.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.8","timestamp":"2020-07-13T06:15:25.431105188Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OXxKeva7ZQcmEiQXaZBhcee","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.435395_96.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.8","timestamp":"2020-07-13T06:15:25.435395684Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OXxKeva7ZQcmEiQXaZBhcee","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.440261_97.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:25.440261819Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IbupanJOb1vHvGDroO9OyV1","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.444849_98.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:25.444849981Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IbupanJOb1vHvGDroO9OyV1","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.475021_99.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.9","timestamp":"2020-07-13T06:15:25.475021111Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sws_fFW3wnseWgGw3cH1pX-","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.479177_100.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.9","timestamp":"2020-07-13T06:15:25.479177003Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sws_fFW3wnseWgGw3cH1pX-","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.481306_101.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:25.481306318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Lq9uI5UBhcdqocC2jNTLpQ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.486249_102.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:25.486249916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Lq9uI5UBhcdqocC2jNTLpQ","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.679609_103.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.10","timestamp":"2020-07-13T06:15:25.679609508Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mlzW7asaUOM35pXKhvz5VX2","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.684249_104.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.10","timestamp":"2020-07-13T06:15:25.684249287Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mlzW7asaUOM35pXKhvz5VX2","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.686238_105.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:25.686238812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IRKdAJCiQhoxy_hbM2Zp4rZ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.690770_106.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:25.690770954Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IRKdAJCiQhoxy_hbM2Zp4rZ","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.750744_107.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.11","timestamp":"2020-07-13T06:15:25.750744600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TVxzlNwS0AToOujH53HcOeO","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.755250_108.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.11","timestamp":"2020-07-13T06:15:25.755250926Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TVxzlNwS0AToOujH53HcOeO","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.759191_109.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:25.759191648Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__V6H1e1TkzH83WDQjMnLs3J","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.763788_110.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:25.763788507Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__V6H1e1TkzH83WDQjMnLs3J","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.874919_111.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.12","timestamp":"2020-07-13T06:15:25.874919330Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cppER_j32QTT6jfkUc7OtN9","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.880056_112.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.12","timestamp":"2020-07-13T06:15:25.880056550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cppER_j32QTT6jfkUc7OtN9","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.882580_113.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:25.882580782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1kbjCgsQlkaQET_-cKcofT8","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.887578_114.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:25.887578813Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1kbjCgsQlkaQET_-cKcofT8","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620926.372198_115.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.13","timestamp":"2020-07-13T06:15:26.372198300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AbGEn19L1mso3vjlMICZuzH","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.376648_116.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.13","timestamp":"2020-07-13T06:15:26.376648564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AbGEn19L1mso3vjlMICZuzH","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620926.379137_117.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:26.379137187Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AJ1-E53HupEVn1PaRJLxknN","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.385397_118.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:26.385397048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AJ1-E53HupEVn1PaRJLxknN","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620926.440877_119.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.14","timestamp":"2020-07-13T06:15:26.440877932Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QwRkzlppS0ZPFANJbrxRLAo","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.445773_120.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.14","timestamp":"2020-07-13T06:15:26.445773115Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QwRkzlppS0ZPFANJbrxRLAo","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620926.447974_121.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:26.447974475Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sXT0pkowqKazgGp3FXuQLsA","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.452307_122.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:26.452307296Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sXT0pkowqKazgGp3FXuQLsA","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620926.558326_123.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.15","timestamp":"2020-07-13T06:15:26.558326355Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OrXmOtQEOSJJK03Tf2okhoA","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.563257_124.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.15","timestamp":"2020-07-13T06:15:26.563257650Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OrXmOtQEOSJJK03Tf2okhoA","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620926.565740_125.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:26.565740914Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AF1yTvVufSmQzFmQD6m6PPG","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.570739_126.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:26.570739160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AF1yTvVufSmQzFmQD6m6PPG","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620929.001475_127.1","section":"bucket.instance","name":"cngrzz-7:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.16","timestamp":"2020-07-13T06:15:29.001475256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ng8rN4G68wRQ3eDQZ46UDUy","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.006047_128.1","section":"bucket.instance","name":"cngrzz-7:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.16","timestamp":"2020-07-13T06:15:29.006047645Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ng8rN4G68wRQ3eDQZ46UDUy","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.008050_129.1","section":"bucket","name":"cngrzz-7","timestamp":"2020-07-13T06:15:29.008050687Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.012811_130.1","section":"bucket","name":"cngrzz-7","timestamp":"2020-07-13T06:15:29.012811406Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.036306_131.1","section":"bucket.instance","name":"cngrzz-8:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.17","timestamp":"2020-07-13T06:15:29.036306449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AreU9GaZl3bJuU19nqk4OSY","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.040693_132.1","section":"bucket.instance","name":"cngrzz-8:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.17","timestamp":"2020-07-13T06:15:29.040693341Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AreU9GaZl3bJuU19nqk4OSY","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.042995_133.1","section":"bucket","name":"cngrzz-8","timestamp":"2020-07-13T06:15:29.042995034Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.047392_134.1","section":"bucket","name":"cngrzz-8","timestamp":"2020-07-13T06:15:29.047392137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.090400_135.1","section":"bucket.instance","name":"cngrzz-9:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.18","timestamp":"2020-07-13T06:15:29.090400277Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k8n1zW2nPOQWnZG2f7x7kJc","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.094794_136.1","section":"bucket.instance","name":"cngrzz-9:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.18","timestamp":"2020-07-13T06:15:29.094794643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k8n1zW2nPOQWnZG2f7x7kJc","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.096824_137.1","section":"bucket","name":"cngrzz-9","timestamp":"2020-07-13T06:15:29.096824917Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.100492_138.1","section":"bucket","name":"cngrzz-9","timestamp":"2020-07-13T06:15:29.100492014Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620932.680034_139.1","section":"bucket","name":"cngrzz-7","timestamp":"2020-07-13T06:15:32.680034714Z","data":{"read_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":1},"write_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":2},"status":{"status":"remove"}}},{"id":"1_1594620932.686075_140.1","section":"bucket","name":"cngrzz-7","timestamp":"2020-07-13T06:15:32.686075353Z","data":{"read_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":1},"write_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":2},"status":{"status":"complete"}}},{"id":"1_1594620933.010327_141.1","section":"bucket","name":"cngrzz-8","timestamp":"2020-07-13T06:15:33.010327021Z","data":{"read_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":1},"write_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":2},"status":{"status":"remove"}}},{"id":"1_1594620933.017231_142.1","section":"bucket","name":"cngrzz-8","timestamp":"2020-07-13T06:15:33.017231115Z","data":{"read_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":1},"write_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":2},"status":{"status":"complete"}}},{"id":"1_1594620933.436438_143.1","section":"bucket","name":"cngrzz-9","timestamp":"2020-07-13T06:15:33.436438596Z","data":{"read_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":1},"write_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":2},"status":{"status":"remove"}}},{"id":"1_1594620933.443793_144.1","section":"bucket","name":"cngrzz-9","timestamp":"2020-07-13T06:15:33.443793026Z","data":{"read_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":1},"write_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":2},"status":{"status":"complete"}}}] 2020-07-13T06:27:05.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '587bd98b-69d2-49ba-9dbd-2be72177aade', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:05.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '587bd98b-69d2-49ba-9dbd-2be72177aade', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:05.412 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 587bd98b-69d2-49ba-9dbd-2be72177aade --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:05.451 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:05.451 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:05.715 INFO:teuthology.orchestra.run.smithi014.stdout:[{"id":"1_1594620792.560320_57.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-07-13T06:13:12.560320432Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w5Jia4Y6RKNUd9E9jGppGgY","ver":1},"status":{"status":"write"}}},{"id":"1_1594620792.569571_63.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-07-13T06:13:12.569571770Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w5Jia4Y6RKNUd9E9jGppGgY","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.353592_71.1","section":"bucket.instance","name":"cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19","timestamp":"2020-07-13T06:15:35.353592485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5YWV65SHwhZo1_hTFsOthdm","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.357969_84.1","section":"bucket.instance","name":"cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19","timestamp":"2020-07-13T06:15:35.357969060Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5YWV65SHwhZo1_hTFsOthdm","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.360453_85.1","section":"bucket","name":"cngrzz-10","timestamp":"2020-07-13T06:15:35.360453117Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lyi7_9F-fQavvS6ttQb4QqQ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.365063_86.1","section":"bucket","name":"cngrzz-10","timestamp":"2020-07-13T06:15:35.365063336Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lyi7_9F-fQavvS6ttQb4QqQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.387630_87.1","section":"bucket.instance","name":"cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20","timestamp":"2020-07-13T06:15:35.387630766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XJOAFF_eewRAyz7Ek55C2eZ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.393964_88.1","section":"bucket.instance","name":"cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20","timestamp":"2020-07-13T06:15:35.393964283Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XJOAFF_eewRAyz7Ek55C2eZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.396127_89.1","section":"bucket","name":"cngrzz-11","timestamp":"2020-07-13T06:15:35.396127319Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uYUbUU7eUol0NqPi_EWhoBS","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.401731_90.1","section":"bucket","name":"cngrzz-11","timestamp":"2020-07-13T06:15:35.401731762Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uYUbUU7eUol0NqPi_EWhoBS","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.439458_91.1","section":"bucket.instance","name":"cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21","timestamp":"2020-07-13T06:15:35.439458898Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y5EI8u7QgEs_npG_hSdZdhk","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.447280_92.1","section":"bucket.instance","name":"cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21","timestamp":"2020-07-13T06:15:35.447280663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y5EI8u7QgEs_npG_hSdZdhk","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.451453_93.1","section":"bucket","name":"cngrzz-12","timestamp":"2020-07-13T06:15:35.451453816Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1xVfXDesczYGyPwaO94C87T","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.457356_94.1","section":"bucket","name":"cngrzz-12","timestamp":"2020-07-13T06:15:35.457356063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1xVfXDesczYGyPwaO94C87T","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.556299_103.1","section":"bucket.instance","name":"cngrzz-13:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.22","timestamp":"2020-07-13T06:15:53.556299812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqQn6_rjhfsJuMkDSydnu0U","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.563929_114.1","section":"bucket.instance","name":"cngrzz-13:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.22","timestamp":"2020-07-13T06:15:53.563929704Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqQn6_rjhfsJuMkDSydnu0U","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.567882_115.1","section":"bucket","name":"cngrzz-13","timestamp":"2020-07-13T06:15:53.567882599Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_raNSztBe-9dO0DB5H_S-No6","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.575383_116.1","section":"bucket","name":"cngrzz-13","timestamp":"2020-07-13T06:15:53.575383568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_raNSztBe-9dO0DB5H_S-No6","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.610132_118.1","section":"bucket.instance","name":"cngrzz-14:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.23","timestamp":"2020-07-13T06:15:53.610132033Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uUpk1HTypA7roPDy4mLwqnM","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.614384_119.1","section":"bucket.instance","name":"cngrzz-14:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.23","timestamp":"2020-07-13T06:15:53.614384828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uUpk1HTypA7roPDy4mLwqnM","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.616654_120.1","section":"bucket","name":"cngrzz-14","timestamp":"2020-07-13T06:15:53.616654760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9zhZRNYkovqDEm3EDgLOk8P","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.622281_121.1","section":"bucket","name":"cngrzz-14","timestamp":"2020-07-13T06:15:53.622281110Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9zhZRNYkovqDEm3EDgLOk8P","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.669597_122.1","section":"bucket.instance","name":"cngrzz-15:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.24","timestamp":"2020-07-13T06:15:53.669597829Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_80eR23_oL7sRegm5pYbqCb3","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.675879_123.1","section":"bucket.instance","name":"cngrzz-15:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.24","timestamp":"2020-07-13T06:15:53.675879396Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_80eR23_oL7sRegm5pYbqCb3","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.677770_124.1","section":"bucket","name":"cngrzz-15","timestamp":"2020-07-13T06:15:53.677770571Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0h_SR2fNa6Li7p9B9ibEN6V","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.682997_125.1","section":"bucket","name":"cngrzz-15","timestamp":"2020-07-13T06:15:53.682997873Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0h_SR2fNa6Li7p9B9ibEN6V","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.903801_180.1","section":"bucket.instance","name":"cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25","timestamp":"2020-07-13T06:16:35.903801109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.908369_187.1","section":"bucket.instance","name":"cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25","timestamp":"2020-07-13T06:16:35.908369870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.910031_188.1","section":"bucket","name":"cngrzz-16","timestamp":"2020-07-13T06:16:35.910031524Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O60kezSGyfXCzbn8fOxwPfi","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.913589_189.1","section":"bucket","name":"cngrzz-16","timestamp":"2020-07-13T06:16:35.913589491Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O60kezSGyfXCzbn8fOxwPfi","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.938308_190.1","section":"bucket.instance","name":"cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26","timestamp":"2020-07-13T06:16:35.938308264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.942759_191.1","section":"bucket.instance","name":"cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26","timestamp":"2020-07-13T06:16:35.942759365Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.944786_192.1","section":"bucket","name":"cngrzz-17","timestamp":"2020-07-13T06:16:35.944786921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uHYXM5KyUTxZhRfiZctWD4n","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.948764_193.1","section":"bucket","name":"cngrzz-17","timestamp":"2020-07-13T06:16:35.948764599Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uHYXM5KyUTxZhRfiZctWD4n","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.990105_194.1","section":"bucket.instance","name":"cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27","timestamp":"2020-07-13T06:16:35.990105813Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.994448_195.1","section":"bucket.instance","name":"cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27","timestamp":"2020-07-13T06:16:35.994448116Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.996481_196.1","section":"bucket","name":"cngrzz-18","timestamp":"2020-07-13T06:16:35.996481980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z5ZtESqqP15O0oomxjuHPqh","ver":1},"status":{"status":"write"}}},{"id":"1_1594620996.000544_197.1","section":"bucket","name":"cngrzz-18","timestamp":"2020-07-13T06:16:36.000544311Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z5ZtESqqP15O0oomxjuHPqh","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620996.023617_198.1","section":"bucket.instance","name":"cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25","timestamp":"2020-07-13T06:16:36.023617389Z","data":{"read_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":1},"write_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":2},"status":{"status":"write"}}},{"id":"1_1594620996.035328_199.1","section":"bucket.instance","name":"cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25","timestamp":"2020-07-13T06:16:36.035328199Z","data":{"read_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":1},"write_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":2},"status":{"status":"complete"}}},{"id":"1_1594620996.042162_200.1","section":"bucket.instance","name":"cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26","timestamp":"2020-07-13T06:16:36.042162134Z","data":{"read_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":1},"write_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":2},"status":{"status":"write"}}},{"id":"1_1594620996.046721_201.1","section":"bucket.instance","name":"cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26","timestamp":"2020-07-13T06:16:36.046721459Z","data":{"read_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":1},"write_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":2},"status":{"status":"complete"}}},{"id":"1_1594620996.060062_202.1","section":"bucket.instance","name":"cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27","timestamp":"2020-07-13T06:16:36.060062705Z","data":{"read_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":1},"write_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":2},"status":{"status":"write"}}},{"id":"1_1594620996.066494_203.1","section":"bucket.instance","name":"cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27","timestamp":"2020-07-13T06:16:36.066494340Z","data":{"read_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":1},"write_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621170.096466_265.1","section":"bucket.instance","name":"cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28","timestamp":"2020-07-13T06:19:30.096466874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":1},"status":{"status":"write"}}},{"id":"1_1594621170.101095_286.1","section":"bucket.instance","name":"cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28","timestamp":"2020-07-13T06:19:30.101095792Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621170.103270_287.1","section":"bucket","name":"cngrzz-19","timestamp":"2020-07-13T06:19:30.103270825Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wQI4zfCNmpgzGLmtq7JYIcF","ver":1},"status":{"status":"write"}}},{"id":"1_1594621170.107053_288.1","section":"bucket","name":"cngrzz-19","timestamp":"2020-07-13T06:19:30.107053912Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wQI4zfCNmpgzGLmtq7JYIcF","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621170.117756_289.1","section":"bucket.instance","name":"cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28","timestamp":"2020-07-13T06:19:30.117756676Z","data":{"read_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":1},"write_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":2},"status":{"status":"write"}}},{"id":"1_1594621170.123767_290.1","section":"bucket.instance","name":"cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28","timestamp":"2020-07-13T06:19:30.123767219Z","data":{"read_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":1},"write_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621322.052070_335.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:22:02.052070538Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":1},"status":{"status":"write"}}},{"id":"1_1594621322.057256_350.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:22:02.057256567Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621322.059913_351.1","section":"bucket","name":"cngrzz-20","timestamp":"2020-07-13T06:22:02.059913372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SiUnqH4drkKjEAXU4lM6KVi","ver":1},"status":{"status":"write"}}},{"id":"1_1594621322.065336_352.1","section":"bucket","name":"cngrzz-20","timestamp":"2020-07-13T06:22:02.065336451Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SiUnqH4drkKjEAXU4lM6KVi","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621329.476722_365.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:22:09.476722756Z","data":{"read_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":1},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":2},"status":{"status":"write"}}},{"id":"1_1594621329.487977_368.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:22:09.487977095Z","data":{"read_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":1},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621390.911893_402.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:23:10.911893325Z","data":{"read_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":2},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":3},"status":{"status":"write"}}},{"id":"1_1594621390.921870_405.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:23:10.921870742Z","data":{"read_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":2},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":3},"status":{"status":"complete"}}},{"id":"1_1594621433.317645_430.1","section":"bucket.instance","name":"cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30","timestamp":"2020-07-13T06:23:53.317645856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":1},"status":{"status":"write"}}},{"id":"1_1594621433.322934_433.1","section":"bucket.instance","name":"cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30","timestamp":"2020-07-13T06:23:53.322934869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621433.324814_434.1","section":"bucket","name":"cngrzz-21","timestamp":"2020-07-13T06:23:53.324814398Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vqw1RgtzigBqWBzMKd9qNek","ver":1},"status":{"status":"write"}}},{"id":"1_1594621433.328408_435.1","section":"bucket","name":"cngrzz-21","timestamp":"2020-07-13T06:23:53.328408882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vqw1RgtzigBqWBzMKd9qNek","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621433.386628_436.1","section":"bucket.instance","name":"cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31","timestamp":"2020-07-13T06:23:53.386628999Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":1},"status":{"status":"write"}}},{"id":"1_1594621433.390873_437.1","section":"bucket.instance","name":"cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31","timestamp":"2020-07-13T06:23:53.390873686Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621433.393237_438.1","section":"bucket","name":"cngrzz-22","timestamp":"2020-07-13T06:23:53.393237663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7bwonnuvmzNIIPT8mR6P0bA","ver":1},"status":{"status":"write"}}},{"id":"1_1594621433.397692_439.1","section":"bucket","name":"cngrzz-22","timestamp":"2020-07-13T06:23:53.397692663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7bwonnuvmzNIIPT8mR6P0bA","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621434.232311_440.1","section":"bucket.instance","name":"cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32","timestamp":"2020-07-13T06:23:54.232311850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":1},"status":{"status":"write"}}},{"id":"1_1594621434.236794_441.1","section":"bucket.instance","name":"cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32","timestamp":"2020-07-13T06:23:54.236794211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621434.238800_442.1","section":"bucket","name":"cngrzz-23","timestamp":"2020-07-13T06:23:54.238800428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SvlhobBY9Slsbwne1CTEC-L","ver":1},"status":{"status":"write"}}},{"id":"1_1594621434.242788_443.1","section":"bucket","name":"cngrzz-23","timestamp":"2020-07-13T06:23:54.242788736Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SvlhobBY9Slsbwne1CTEC-L","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621434.264342_444.1","section":"bucket.instance","name":"cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30","timestamp":"2020-07-13T06:23:54.264342494Z","data":{"read_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":1},"write_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":2},"status":{"status":"write"}}},{"id":"1_1594621434.270547_445.1","section":"bucket.instance","name":"cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30","timestamp":"2020-07-13T06:23:54.270547881Z","data":{"read_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":1},"write_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621434.275368_446.1","section":"bucket.instance","name":"cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31","timestamp":"2020-07-13T06:23:54.275368951Z","data":{"read_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":1},"write_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":2},"status":{"status":"write"}}},{"id":"1_1594621434.282613_447.1","section":"bucket.instance","name":"cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31","timestamp":"2020-07-13T06:23:54.282613618Z","data":{"read_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":1},"write_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621434.294881_448.1","section":"bucket.instance","name":"cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32","timestamp":"2020-07-13T06:23:54.294881064Z","data":{"read_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":1},"write_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":2},"status":{"status":"write"}}},{"id":"1_1594621434.303397_449.1","section":"bucket.instance","name":"cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32","timestamp":"2020-07-13T06:23:54.303397214Z","data":{"read_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":1},"write_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621455.307113_481.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.307113064Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.311789_492.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.311789146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.314390_493.1","section":"bucket","name":"cngrzz-24","timestamp":"2020-07-13T06:24:15.314390695Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_syYOTmiVtbmFjOCG8p3NuzV","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.318379_494.1","section":"bucket","name":"cngrzz-24","timestamp":"2020-07-13T06:24:15.318379915Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_syYOTmiVtbmFjOCG8p3NuzV","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.344614_495.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.344614344Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.349148_496.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.349148544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.351560_497.1","section":"bucket","name":"cngrzz-25","timestamp":"2020-07-13T06:24:15.351560923Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L6eQ90-G-twMkpkPL-m5q_J","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.355886_498.1","section":"bucket","name":"cngrzz-25","timestamp":"2020-07-13T06:24:15.355886460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L6eQ90-G-twMkpkPL-m5q_J","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.405945_499.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.405945564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.410863_500.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.410863468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.412859_501.1","section":"bucket","name":"cngrzz-26","timestamp":"2020-07-13T06:24:15.412859756Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wVuwzieS4x5WATempXhxurc","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.417215_502.1","section":"bucket","name":"cngrzz-26","timestamp":"2020-07-13T06:24:15.417215461Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wVuwzieS4x5WATempXhxurc","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.441624_503.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.441624712Z","data":{"read_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":1},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":2},"status":{"status":"write"}}},{"id":"1_1594621455.464024_504.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.464024694Z","data":{"read_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":1},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621455.470869_505.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.470869735Z","data":{"read_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":2},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":3},"status":{"status":"write"}}},{"id":"1_1594621455.475511_506.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.475511044Z","data":{"read_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":2},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":3},"status":{"status":"complete"}}},{"id":"1_1594621455.479588_507.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.479588113Z","data":{"read_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":1},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":2},"status":{"status":"write"}}},{"id":"1_1594621455.484136_508.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.484136677Z","data":{"read_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":1},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621455.494806_509.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.494806032Z","data":{"read_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":2},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":3},"status":{"status":"write"}}},{"id":"1_1594621455.505325_510.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.505325897Z","data":{"read_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":2},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":3},"status":{"status":"complete"}}},{"id":"1_1594621455.518131_511.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.518131893Z","data":{"read_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":1},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":2},"status":{"status":"write"}}},{"id":"1_1594621455.527513_512.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.527513994Z","data":{"read_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":1},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621455.542157_513.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.542157223Z","data":{"read_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":2},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":3},"status":{"status":"write"}}},{"id":"1_1594621455.551984_514.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.551984028Z","data":{"read_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":2},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":3},"status":{"status":"complete"}}},{"id":"1_1594621478.463758_549.1","section":"bucket.instance","name":"cngrzz-27:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.36","timestamp":"2020-07-13T06:24:38.463758191Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.468096_556.1","section":"bucket.instance","name":"cngrzz-27:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.36","timestamp":"2020-07-13T06:24:38.468096388Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.470142_557.1","section":"bucket","name":"cngrzz-27","timestamp":"2020-07-13T06:24:38.470142675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xKmuAPoskvLbXPU03OlC73s","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.473856_558.1","section":"bucket","name":"cngrzz-27","timestamp":"2020-07-13T06:24:38.473856655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xKmuAPoskvLbXPU03OlC73s","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.497896_559.1","section":"bucket.instance","name":"cngrzz-28:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.37","timestamp":"2020-07-13T06:24:38.497896309Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.502362_560.1","section":"bucket.instance","name":"cngrzz-28:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.37","timestamp":"2020-07-13T06:24:38.502362092Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.504706_561.1","section":"bucket","name":"cngrzz-28","timestamp":"2020-07-13T06:24:38.504706153Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m0vuxwdxcV4K0zwl3eB9Wa9","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.509299_562.1","section":"bucket","name":"cngrzz-28","timestamp":"2020-07-13T06:24:38.509299001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m0vuxwdxcV4K0zwl3eB9Wa9","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.554287_563.1","section":"bucket.instance","name":"cngrzz-29:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.38","timestamp":"2020-07-13T06:24:38.554287987Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.561718_564.1","section":"bucket.instance","name":"cngrzz-29:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.38","timestamp":"2020-07-13T06:24:38.561718878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.564680_565.1","section":"bucket","name":"cngrzz-29","timestamp":"2020-07-13T06:24:38.564680554Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2TuYKigZrzuv-dMOyxValfc","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.569842_566.1","section":"bucket","name":"cngrzz-29","timestamp":"2020-07-13T06:24:38.569842326Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2TuYKigZrzuv-dMOyxValfc","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.595037_567.1","section":"bucket.instance","name":"cngrzz-27:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.36","timestamp":"2020-07-13T06:24:38.595037740Z","data":{"read_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":1},"write_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":2},"status":{"status":"write"}}},{"id":"1_1594621478.600762_568.1","section":"bucket.instance","name":"cngrzz-27:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.36","timestamp":"2020-07-13T06:24:38.600762996Z","data":{"read_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":1},"write_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621478.647960_569.1","section":"bucket.instance","name":"cngrzz-28:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.37","timestamp":"2020-07-13T06:24:38.647960510Z","data":{"read_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":1},"write_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":2},"status":{"status":"write"}}},{"id":"1_1594621478.658940_570.1","section":"bucket.instance","name":"cngrzz-28:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.37","timestamp":"2020-07-13T06:24:38.658940233Z","data":{"read_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":1},"write_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621478.723174_571.1","section":"bucket.instance","name":"cngrzz-29:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.38","timestamp":"2020-07-13T06:24:38.723174021Z","data":{"read_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":1},"write_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":2},"status":{"status":"write"}}},{"id":"1_1594621478.729364_572.1","section":"bucket.instance","name":"cngrzz-29:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.38","timestamp":"2020-07-13T06:24:38.729364605Z","data":{"read_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":1},"write_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621478.803680_573.1","section":"bucket.instance","name":"cngrzz-30:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.39","timestamp":"2020-07-13T06:24:38.803680169Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.808550_574.1","section":"bucket.instance","name":"cngrzz-30:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.39","timestamp":"2020-07-13T06:24:38.808550868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.810788_575.1","section":"bucket","name":"cngrzz-30","timestamp":"2020-07-13T06:24:38.810788109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tcbDPAiLUbg1Qosf4aVJZaR","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.814964_576.1","section":"bucket","name":"cngrzz-30","timestamp":"2020-07-13T06:24:38.814964209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tcbDPAiLUbg1Qosf4aVJZaR","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.842441_577.1","section":"bucket.instance","name":"cngrzz-31:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.40","timestamp":"2020-07-13T06:24:38.842441698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.846740_578.1","section":"bucket.instance","name":"cngrzz-31:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.40","timestamp":"2020-07-13T06:24:38.846740063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.849242_579.1","section":"bucket","name":"cngrzz-31","timestamp":"2020-07-13T06:24:38.849242603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PcNMXYzKJiqzUL9v_g9UUEW","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.853963_580.1","section":"bucket","name":"cngrzz-31","timestamp":"2020-07-13T06:24:38.853963184Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PcNMXYzKJiqzUL9v_g9UUEW","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.938346_581.1","section":"bucket.instance","name":"cngrzz-32:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.41","timestamp":"2020-07-13T06:24:38.938346254Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.942593_582.1","section":"bucket.instance","name":"cngrzz-32:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.41","timestamp":"2020-07-13T06:24:38.942593554Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.944644_583.1","section":"bucket","name":"cngrzz-32","timestamp":"2020-07-13T06:24:38.944644272Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cgRpl2sxiftnVUa2OO_gEgo","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.948537_584.1","section":"bucket","name":"cngrzz-32","timestamp":"2020-07-13T06:24:38.948537812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cgRpl2sxiftnVUa2OO_gEgo","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.974066_585.1","section":"bucket.instance","name":"cngrzz-30:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.39","timestamp":"2020-07-13T06:24:38.974066414Z","data":{"read_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":1},"write_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":2},"status":{"status":"write"}}},{"id":"1_1594621478.980780_586.1","section":"bucket.instance","name":"cngrzz-30:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.39","timestamp":"2020-07-13T06:24:38.980780610Z","data":{"read_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":1},"write_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621479.033952_587.1","section":"bucket.instance","name":"cngrzz-31:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.40","timestamp":"2020-07-13T06:24:39.033952236Z","data":{"read_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":1},"write_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":2},"status":{"status":"write"}}},{"id":"1_1594621479.045331_588.1","section":"bucket.instance","name":"cngrzz-31:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.40","timestamp":"2020-07-13T06:24:39.045331654Z","data":{"read_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":1},"write_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621479.109134_589.1","section":"bucket.instance","name":"cngrzz-32:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.41","timestamp":"2020-07-13T06:24:39.109134981Z","data":{"read_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":1},"write_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":2},"status":{"status":"write"}}},{"id":"1_1594621479.118646_590.1","section":"bucket.instance","name":"cngrzz-32:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.41","timestamp":"2020-07-13T06:24:39.118646708Z","data":{"read_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":1},"write_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621479.194094_591.1","section":"bucket.instance","name":"cngrzz-33:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.42","timestamp":"2020-07-13T06:24:39.194094933Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.200722_592.1","section":"bucket.instance","name":"cngrzz-33:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.42","timestamp":"2020-07-13T06:24:39.200722310Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.203027_593.1","section":"bucket","name":"cngrzz-33","timestamp":"2020-07-13T06:24:39.203027753Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PuawPXsI3tXqBy_MXXvr0Cy","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.209920_594.1","section":"bucket","name":"cngrzz-33","timestamp":"2020-07-13T06:24:39.209920207Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PuawPXsI3tXqBy_MXXvr0Cy","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.241265_595.1","section":"bucket.instance","name":"cngrzz-34:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.43","timestamp":"2020-07-13T06:24:39.241265494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.246173_596.1","section":"bucket.instance","name":"cngrzz-34:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.43","timestamp":"2020-07-13T06:24:39.246173690Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.248646_597.1","section":"bucket","name":"cngrzz-34","timestamp":"2020-07-13T06:24:39.248646104Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SZllY3UvM6YWJqOBtYiiacb","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.253617_598.1","section":"bucket","name":"cngrzz-34","timestamp":"2020-07-13T06:24:39.253617145Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SZllY3UvM6YWJqOBtYiiacb","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.323567_599.1","section":"bucket.instance","name":"cngrzz-35:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.44","timestamp":"2020-07-13T06:24:39.323567314Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.328276_600.1","section":"bucket.instance","name":"cngrzz-35:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.44","timestamp":"2020-07-13T06:24:39.328276870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.330806_601.1","section":"bucket","name":"cngrzz-35","timestamp":"2020-07-13T06:24:39.330806173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v-R90_jF6GVW73YIo8cHfq0","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.336092_602.1","section":"bucket","name":"cngrzz-35","timestamp":"2020-07-13T06:24:39.336092549Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v-R90_jF6GVW73YIo8cHfq0","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.359801_603.1","section":"bucket.instance","name":"cngrzz-33:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.42","timestamp":"2020-07-13T06:24:39.359801153Z","data":{"read_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":1},"write_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":2},"status":{"status":"write"}}},{"id":"1_1594621479.369370_604.1","section":"bucket.instance","name":"cngrzz-33:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.42","timestamp":"2020-07-13T06:24:39.369370295Z","data":{"read_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":1},"write_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621480.420657_605.1","section":"bucket.instance","name":"cngrzz-34:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.43","timestamp":"2020-07-13T06:24:40.420657898Z","data":{"read_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":1},"write_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":2},"status":{"status":"write"}}},{"id":"1_1594621480.432601_606.1","section":"bucket.instance","name":"cngrzz-34:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.43","timestamp":"2020-07-13T06:24:40.432601102Z","data":{"read_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":1},"write_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621482.474186_607.1","section":"bucket.instance","name":"cngrzz-35:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.44","timestamp":"2020-07-13T06:24:42.474186915Z","data":{"read_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":1},"write_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":2},"status":{"status":"write"}}},{"id":"1_1594621482.484822_608.1","section":"bucket.instance","name":"cngrzz-35:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.44","timestamp":"2020-07-13T06:24:42.484822401Z","data":{"read_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":1},"write_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621482.913401_609.1","section":"bucket.instance","name":"cngrzz-36:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.45","timestamp":"2020-07-13T06:24:42.913401257Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BHYIPGqLECVNEi2cZ-MYjOz","ver":1},"status":{"status":"write"}}},{"id":"1_1594621482.917800_610.1","section":"bucket.instance","name":"cngrzz-36:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.45","timestamp":"2020-07-13T06:24:42.917800056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BHYIPGqLECVNEi2cZ-MYjOz","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621482.920043_611.1","section":"bucket","name":"cngrzz-36","timestamp":"2020-07-13T06:24:42.920043693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lDf1FKkxnRubfsQ9M1--dgD","ver":1},"status":{"status":"write"}}},{"id":"1_1594621482.924506_612.1","section":"bucket","name":"cngrzz-36","timestamp":"2020-07-13T06:24:42.924506470Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lDf1FKkxnRubfsQ9M1--dgD","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621483.015114_613.1","section":"bucket.instance","name":"cngrzz-37:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.46","timestamp":"2020-07-13T06:24:43.015114675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oP_SUc8PxL6c4XMdnGrcIDd","ver":1},"status":{"status":"write"}}},{"id":"1_1594621483.020538_614.1","section":"bucket.instance","name":"cngrzz-37:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.46","timestamp":"2020-07-13T06:24:43.020538172Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oP_SUc8PxL6c4XMdnGrcIDd","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621483.022545_615.1","section":"bucket","name":"cngrzz-37","timestamp":"2020-07-13T06:24:43.022545950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5gFmRJU_Lnfca_c3ktRTso7","ver":1},"status":{"status":"write"}}},{"id":"1_1594621483.026371_616.1","section":"bucket","name":"cngrzz-37","timestamp":"2020-07-13T06:24:43.026371722Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5gFmRJU_Lnfca_c3ktRTso7","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621483.081477_617.1","section":"bucket.instance","name":"cngrzz-38:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.47","timestamp":"2020-07-13T06:24:43.081477070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nVm3w2ql0Kn2oqc2UsEJH8a","ver":1},"status":{"status":"write"}}},{"id":"1_1594621483.087242_618.1","section":"bucket.instance","name":"cngrzz-38:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.47","timestamp":"2020-07-13T06:24:43.087242473Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nVm3w2ql0Kn2oqc2UsEJH8a","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621483.089407_619.1","section":"bucket","name":"cngrzz-38","timestamp":"2020-07-13T06:24:43.089407632Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3eyVOL0MMcQBz9oFqM87zxz","ver":1},"status":{"status":"write"}}},{"id":"1_1594621483.093117_620.1","section":"bucket","name":"cngrzz-38","timestamp":"2020-07-13T06:24:43.093117980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3eyVOL0MMcQBz9oFqM87zxz","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.461368_621.1","section":"bucket.instance","name":"cngrzz-39:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.48","timestamp":"2020-07-13T06:24:46.461368112Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uMnm_CbH26xOW2j2VokpY4s","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.466851_624.1","section":"bucket.instance","name":"cngrzz-39:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.48","timestamp":"2020-07-13T06:24:46.466851306Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uMnm_CbH26xOW2j2VokpY4s","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.469013_625.1","section":"bucket","name":"cngrzz-39","timestamp":"2020-07-13T06:24:46.469013868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eqEaW0o9d4QYozEF28u7c49","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.472599_626.1","section":"bucket","name":"cngrzz-39","timestamp":"2020-07-13T06:24:46.472599369Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eqEaW0o9d4QYozEF28u7c49","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.759727_627.1","section":"bucket.instance","name":"cngrzz-40:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.49","timestamp":"2020-07-13T06:24:46.759727960Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OP3txqbvev88w0dP9vK-dO5","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.765631_628.1","section":"bucket.instance","name":"cngrzz-40:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.49","timestamp":"2020-07-13T06:24:46.765631411Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OP3txqbvev88w0dP9vK-dO5","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.767996_629.1","section":"bucket","name":"cngrzz-40","timestamp":"2020-07-13T06:24:46.767996150Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M3LKRjOIblMfECEncxsHzyu","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.772433_630.1","section":"bucket","name":"cngrzz-40","timestamp":"2020-07-13T06:24:46.772433975Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M3LKRjOIblMfECEncxsHzyu","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.817605_631.1","section":"bucket.instance","name":"cngrzz-41:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.50","timestamp":"2020-07-13T06:24:46.817605518Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kHTWaobdOcV3WvsIXS-4JDw","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.823059_632.1","section":"bucket.instance","name":"cngrzz-41:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.50","timestamp":"2020-07-13T06:24:46.823059818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kHTWaobdOcV3WvsIXS-4JDw","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.825455_633.1","section":"bucket","name":"cngrzz-41","timestamp":"2020-07-13T06:24:46.825455311Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_enzpzPLLpAzWWe8NZI6ZjKh","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.829707_634.1","section":"bucket","name":"cngrzz-41","timestamp":"2020-07-13T06:24:46.829707023Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_enzpzPLLpAzWWe8NZI6ZjKh","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620914.565843_67.1","section":"bucket.instance","name":"cngrzz-1:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.1","timestamp":"2020-07-13T06:15:14.565843290Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NrX7NUBlKlctPxLgZQhVZeJ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620914.569624_68.1","section":"bucket.instance","name":"cngrzz-1:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.1","timestamp":"2020-07-13T06:15:14.569624562Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NrX7NUBlKlctPxLgZQhVZeJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620914.571621_69.1","section":"bucket","name":"cngrzz-1","timestamp":"2020-07-13T06:15:14.571621151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B_xEGfjqp5GJf-kckzrymxy","ver":1},"status":{"status":"write"}}},{"id":"1_1594620914.576799_70.1","section":"bucket","name":"cngrzz-1","timestamp":"2020-07-13T06:15:14.576799306Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B_xEGfjqp5GJf-kckzrymxy","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620914.604902_71.1","section":"bucket.instance","name":"cngrzz-2:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.2","timestamp":"2020-07-13T06:15:14.604902219Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PvZsoXHBEggBhfEef7o5U0T","ver":1},"status":{"status":"write"}}},{"id":"1_1594620914.609491_72.1","section":"bucket.instance","name":"cngrzz-2:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.2","timestamp":"2020-07-13T06:15:14.609491570Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PvZsoXHBEggBhfEef7o5U0T","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620914.612054_73.1","section":"bucket","name":"cngrzz-2","timestamp":"2020-07-13T06:15:14.612054698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6IT0MMiYAzd8ADA6mJQGl2V","ver":1},"status":{"status":"write"}}},{"id":"1_1594620914.620627_74.1","section":"bucket","name":"cngrzz-2","timestamp":"2020-07-13T06:15:14.620627832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6IT0MMiYAzd8ADA6mJQGl2V","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620918.616085_75.1","section":"bucket.instance","name":"cngrzz-3:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.3","timestamp":"2020-07-13T06:15:18.616085899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jNBNF8gIfCQ0VtU1Nd6VsXo","ver":1},"status":{"status":"write"}}},{"id":"1_1594620918.620674_76.1","section":"bucket.instance","name":"cngrzz-3:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.3","timestamp":"2020-07-13T06:15:18.620674736Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jNBNF8gIfCQ0VtU1Nd6VsXo","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620918.625187_77.1","section":"bucket","name":"cngrzz-3","timestamp":"2020-07-13T06:15:18.625187717Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6UxFHip67VoBBxh6FLU5ArZ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620918.629897_78.1","section":"bucket","name":"cngrzz-3","timestamp":"2020-07-13T06:15:18.629897499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6UxFHip67VoBBxh6FLU5ArZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.868689_79.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.4","timestamp":"2020-07-13T06:15:21.868689028Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5eRMS50W9omalXkI-eKEkZH","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.873048_80.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.4","timestamp":"2020-07-13T06:15:21.873048233Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5eRMS50W9omalXkI-eKEkZH","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.874813_81.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:21.874813555Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JboyausbemE9wke0D-OFRJL","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.881925_82.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:21.881925098Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JboyausbemE9wke0D-OFRJL","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.904963_83.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.5","timestamp":"2020-07-13T06:15:21.904963978Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_orhbCdPThQDntboRt2h0Sl9","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.910649_84.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.5","timestamp":"2020-07-13T06:15:21.910649089Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_orhbCdPThQDntboRt2h0Sl9","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.913196_85.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:21.913196616Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W9AKCYVnn-UeTSsMsys7MT_","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.920774_86.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:21.920774888Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W9AKCYVnn-UeTSsMsys7MT_","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.970374_87.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.6","timestamp":"2020-07-13T06:15:21.970374493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v6ZB9vPFQIwn4oxtloegK6s","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.976019_88.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.6","timestamp":"2020-07-13T06:15:21.976019891Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v6ZB9vPFQIwn4oxtloegK6s","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.980716_89.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:21.980716496Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IjM-H8ZOw8IgEVRDW0EGVzm","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.991312_90.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:21.991312189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IjM-H8ZOw8IgEVRDW0EGVzm","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.389888_91.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.7","timestamp":"2020-07-13T06:15:25.389888371Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RWdhBH265JDcuhXxt8yJvwV","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.394176_92.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.7","timestamp":"2020-07-13T06:15:25.394176539Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RWdhBH265JDcuhXxt8yJvwV","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.396386_93.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:25.396386821Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t0iFgy3ChisA7Hu7SEB54i8","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.400663_94.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:25.400663561Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t0iFgy3ChisA7Hu7SEB54i8","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.431105_95.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.8","timestamp":"2020-07-13T06:15:25.431105188Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OXxKeva7ZQcmEiQXaZBhcee","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.435395_96.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.8","timestamp":"2020-07-13T06:15:25.435395684Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OXxKeva7ZQcmEiQXaZBhcee","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.440261_97.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:25.440261819Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IbupanJOb1vHvGDroO9OyV1","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.444849_98.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:25.444849981Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IbupanJOb1vHvGDroO9OyV1","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.475021_99.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.9","timestamp":"2020-07-13T06:15:25.475021111Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sws_fFW3wnseWgGw3cH1pX-","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.479177_100.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.9","timestamp":"2020-07-13T06:15:25.479177003Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sws_fFW3wnseWgGw3cH1pX-","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.481306_101.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:25.481306318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Lq9uI5UBhcdqocC2jNTLpQ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.486249_102.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:25.486249916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Lq9uI5UBhcdqocC2jNTLpQ","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.679609_103.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.10","timestamp":"2020-07-13T06:15:25.679609508Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mlzW7asaUOM35pXKhvz5VX2","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.684249_104.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.10","timestamp":"2020-07-13T06:15:25.684249287Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mlzW7asaUOM35pXKhvz5VX2","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.686238_105.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:25.686238812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IRKdAJCiQhoxy_hbM2Zp4rZ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.690770_106.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:25.690770954Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IRKdAJCiQhoxy_hbM2Zp4rZ","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.750744_107.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.11","timestamp":"2020-07-13T06:15:25.750744600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TVxzlNwS0AToOujH53HcOeO","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.755250_108.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.11","timestamp":"2020-07-13T06:15:25.755250926Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TVxzlNwS0AToOujH53HcOeO","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.759191_109.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:25.759191648Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__V6H1e1TkzH83WDQjMnLs3J","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.763788_110.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:25.763788507Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__V6H1e1TkzH83WDQjMnLs3J","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.874919_111.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.12","timestamp":"2020-07-13T06:15:25.874919330Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cppER_j32QTT6jfkUc7OtN9","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.880056_112.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.12","timestamp":"2020-07-13T06:15:25.880056550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cppER_j32QTT6jfkUc7OtN9","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.882580_113.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:25.882580782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1kbjCgsQlkaQET_-cKcofT8","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.887578_114.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:25.887578813Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1kbjCgsQlkaQET_-cKcofT8","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620926.372198_115.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.13","timestamp":"2020-07-13T06:15:26.372198300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AbGEn19L1mso3vjlMICZuzH","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.376648_116.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.13","timestamp":"2020-07-13T06:15:26.376648564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AbGEn19L1mso3vjlMICZuzH","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620926.379137_117.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:26.379137187Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AJ1-E53HupEVn1PaRJLxknN","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.385397_118.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:26.385397048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AJ1-E53HupEVn1PaRJLxknN","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620926.440877_119.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.14","timestamp":"2020-07-13T06:15:26.440877932Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QwRkzlppS0ZPFANJbrxRLAo","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.445773_120.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.14","timestamp":"2020-07-13T06:15:26.445773115Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QwRkzlppS0ZPFANJbrxRLAo","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620926.447974_121.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:26.447974475Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sXT0pkowqKazgGp3FXuQLsA","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.452307_122.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:26.452307296Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sXT0pkowqKazgGp3FXuQLsA","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620926.558326_123.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.15","timestamp":"2020-07-13T06:15:26.558326355Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OrXmOtQEOSJJK03Tf2okhoA","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.563257_124.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.15","timestamp":"2020-07-13T06:15:26.563257650Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OrXmOtQEOSJJK03Tf2okhoA","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620926.565740_125.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:26.565740914Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AF1yTvVufSmQzFmQD6m6PPG","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.570739_126.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:26.570739160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AF1yTvVufSmQzFmQD6m6PPG","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620929.001475_127.1","section":"bucket.instance","name":"cngrzz-7:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.16","timestamp":"2020-07-13T06:15:29.001475256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ng8rN4G68wRQ3eDQZ46UDUy","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.006047_128.1","section":"bucket.instance","name":"cngrzz-7:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.16","timestamp":"2020-07-13T06:15:29.006047645Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ng8rN4G68wRQ3eDQZ46UDUy","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.008050_129.1","section":"bucket","name":"cngrzz-7","timestamp":"2020-07-13T06:15:29.008050687Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.012811_130.1","section":"bucket","name":"cngrzz-7","timestamp":"2020-07-13T06:15:29.012811406Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.036306_131.1","section":"bucket.instance","name":"cngrzz-8:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.17","timestamp":"2020-07-13T06:15:29.036306449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AreU9GaZl3bJuU19nqk4OSY","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.040693_132.1","section":"bucket.instance","name":"cngrzz-8:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.17","timestamp":"2020-07-13T06:15:29.040693341Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AreU9GaZl3bJuU19nqk4OSY","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.042995_133.1","section":"bucket","name":"cngrzz-8","timestamp":"2020-07-13T06:15:29.042995034Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.047392_134.1","section":"bucket","name":"cngrzz-8","timestamp":"2020-07-13T06:15:29.047392137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.090400_135.1","section":"bucket.instance","name":"cngrzz-9:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.18","timestamp":"2020-07-13T06:15:29.090400277Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k8n1zW2nPOQWnZG2f7x7kJc","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.094794_136.1","section":"bucket.instance","name":"cngrzz-9:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.18","timestamp":"2020-07-13T06:15:29.094794643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k8n1zW2nPOQWnZG2f7x7kJc","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.096824_137.1","section":"bucket","name":"cngrzz-9","timestamp":"2020-07-13T06:15:29.096824917Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.100492_138.1","section":"bucket","name":"cngrzz-9","timestamp":"2020-07-13T06:15:29.100492014Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620932.680034_139.1","section":"bucket","name":"cngrzz-7","timestamp":"2020-07-13T06:15:32.680034714Z","data":{"read_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":1},"write_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":2},"status":{"status":"remove"}}},{"id":"1_1594620932.686075_140.1","section":"bucket","name":"cngrzz-7","timestamp":"2020-07-13T06:15:32.686075353Z","data":{"read_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":1},"write_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":2},"status":{"status":"complete"}}},{"id":"1_1594620933.010327_141.1","section":"bucket","name":"cngrzz-8","timestamp":"2020-07-13T06:15:33.010327021Z","data":{"read_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":1},"write_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":2},"status":{"status":"remove"}}},{"id":"1_1594620933.017231_142.1","section":"bucket","name":"cngrzz-8","timestamp":"2020-07-13T06:15:33.017231115Z","data":{"read_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":1},"write_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":2},"status":{"status":"complete"}}},{"id":"1_1594620933.436438_143.1","section":"bucket","name":"cngrzz-9","timestamp":"2020-07-13T06:15:33.436438596Z","data":{"read_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":1},"write_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":2},"status":{"status":"remove"}}},{"id":"1_1594620933.443793_144.1","section":"bucket","name":"cngrzz-9","timestamp":"2020-07-13T06:15:33.443793026Z","data":{"read_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":1},"write_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":2},"status":{"status":"complete"}}}] 2020-07-13T06:27:05.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '587bd98b-69d2-49ba-9dbd-2be72177aade', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:05.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '587bd98b-69d2-49ba-9dbd-2be72177aade', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:05.719 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 587bd98b-69d2-49ba-9dbd-2be72177aade --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:05.759 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:05.760 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:06.054 INFO:teuthology.orchestra.run.smithi017.stdout:[{"id":"1_1594620792.560320_57.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-07-13T06:13:12.560320432Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w5Jia4Y6RKNUd9E9jGppGgY","ver":1},"status":{"status":"write"}}},{"id":"1_1594620792.569571_63.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-07-13T06:13:12.569571770Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w5Jia4Y6RKNUd9E9jGppGgY","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.353592_71.1","section":"bucket.instance","name":"cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19","timestamp":"2020-07-13T06:15:35.353592485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5YWV65SHwhZo1_hTFsOthdm","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.357969_84.1","section":"bucket.instance","name":"cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19","timestamp":"2020-07-13T06:15:35.357969060Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5YWV65SHwhZo1_hTFsOthdm","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.360453_85.1","section":"bucket","name":"cngrzz-10","timestamp":"2020-07-13T06:15:35.360453117Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lyi7_9F-fQavvS6ttQb4QqQ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.365063_86.1","section":"bucket","name":"cngrzz-10","timestamp":"2020-07-13T06:15:35.365063336Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lyi7_9F-fQavvS6ttQb4QqQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.387630_87.1","section":"bucket.instance","name":"cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20","timestamp":"2020-07-13T06:15:35.387630766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XJOAFF_eewRAyz7Ek55C2eZ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.393964_88.1","section":"bucket.instance","name":"cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20","timestamp":"2020-07-13T06:15:35.393964283Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XJOAFF_eewRAyz7Ek55C2eZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.396127_89.1","section":"bucket","name":"cngrzz-11","timestamp":"2020-07-13T06:15:35.396127319Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uYUbUU7eUol0NqPi_EWhoBS","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.401731_90.1","section":"bucket","name":"cngrzz-11","timestamp":"2020-07-13T06:15:35.401731762Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uYUbUU7eUol0NqPi_EWhoBS","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.439458_91.1","section":"bucket.instance","name":"cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21","timestamp":"2020-07-13T06:15:35.439458898Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y5EI8u7QgEs_npG_hSdZdhk","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.447280_92.1","section":"bucket.instance","name":"cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21","timestamp":"2020-07-13T06:15:35.447280663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y5EI8u7QgEs_npG_hSdZdhk","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.451453_93.1","section":"bucket","name":"cngrzz-12","timestamp":"2020-07-13T06:15:35.451453816Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1xVfXDesczYGyPwaO94C87T","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.457356_94.1","section":"bucket","name":"cngrzz-12","timestamp":"2020-07-13T06:15:35.457356063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1xVfXDesczYGyPwaO94C87T","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.556299_103.1","section":"bucket.instance","name":"cngrzz-13:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.22","timestamp":"2020-07-13T06:15:53.556299812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqQn6_rjhfsJuMkDSydnu0U","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.563929_114.1","section":"bucket.instance","name":"cngrzz-13:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.22","timestamp":"2020-07-13T06:15:53.563929704Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqQn6_rjhfsJuMkDSydnu0U","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.567882_115.1","section":"bucket","name":"cngrzz-13","timestamp":"2020-07-13T06:15:53.567882599Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_raNSztBe-9dO0DB5H_S-No6","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.575383_116.1","section":"bucket","name":"cngrzz-13","timestamp":"2020-07-13T06:15:53.575383568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_raNSztBe-9dO0DB5H_S-No6","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.610132_118.1","section":"bucket.instance","name":"cngrzz-14:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.23","timestamp":"2020-07-13T06:15:53.610132033Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uUpk1HTypA7roPDy4mLwqnM","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.614384_119.1","section":"bucket.instance","name":"cngrzz-14:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.23","timestamp":"2020-07-13T06:15:53.614384828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uUpk1HTypA7roPDy4mLwqnM","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.616654_120.1","section":"bucket","name":"cngrzz-14","timestamp":"2020-07-13T06:15:53.616654760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9zhZRNYkovqDEm3EDgLOk8P","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.622281_121.1","section":"bucket","name":"cngrzz-14","timestamp":"2020-07-13T06:15:53.622281110Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9zhZRNYkovqDEm3EDgLOk8P","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.669597_122.1","section":"bucket.instance","name":"cngrzz-15:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.24","timestamp":"2020-07-13T06:15:53.669597829Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_80eR23_oL7sRegm5pYbqCb3","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.675879_123.1","section":"bucket.instance","name":"cngrzz-15:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.24","timestamp":"2020-07-13T06:15:53.675879396Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_80eR23_oL7sRegm5pYbqCb3","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.677770_124.1","section":"bucket","name":"cngrzz-15","timestamp":"2020-07-13T06:15:53.677770571Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0h_SR2fNa6Li7p9B9ibEN6V","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.682997_125.1","section":"bucket","name":"cngrzz-15","timestamp":"2020-07-13T06:15:53.682997873Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0h_SR2fNa6Li7p9B9ibEN6V","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.903801_180.1","section":"bucket.instance","name":"cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25","timestamp":"2020-07-13T06:16:35.903801109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.908369_187.1","section":"bucket.instance","name":"cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25","timestamp":"2020-07-13T06:16:35.908369870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.910031_188.1","section":"bucket","name":"cngrzz-16","timestamp":"2020-07-13T06:16:35.910031524Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O60kezSGyfXCzbn8fOxwPfi","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.913589_189.1","section":"bucket","name":"cngrzz-16","timestamp":"2020-07-13T06:16:35.913589491Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O60kezSGyfXCzbn8fOxwPfi","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.938308_190.1","section":"bucket.instance","name":"cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26","timestamp":"2020-07-13T06:16:35.938308264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.942759_191.1","section":"bucket.instance","name":"cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26","timestamp":"2020-07-13T06:16:35.942759365Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.944786_192.1","section":"bucket","name":"cngrzz-17","timestamp":"2020-07-13T06:16:35.944786921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uHYXM5KyUTxZhRfiZctWD4n","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.948764_193.1","section":"bucket","name":"cngrzz-17","timestamp":"2020-07-13T06:16:35.948764599Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uHYXM5KyUTxZhRfiZctWD4n","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.990105_194.1","section":"bucket.instance","name":"cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27","timestamp":"2020-07-13T06:16:35.990105813Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.994448_195.1","section":"bucket.instance","name":"cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27","timestamp":"2020-07-13T06:16:35.994448116Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.996481_196.1","section":"bucket","name":"cngrzz-18","timestamp":"2020-07-13T06:16:35.996481980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z5ZtESqqP15O0oomxjuHPqh","ver":1},"status":{"status":"write"}}},{"id":"1_1594620996.000544_197.1","section":"bucket","name":"cngrzz-18","timestamp":"2020-07-13T06:16:36.000544311Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z5ZtESqqP15O0oomxjuHPqh","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620996.023617_198.1","section":"bucket.instance","name":"cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25","timestamp":"2020-07-13T06:16:36.023617389Z","data":{"read_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":1},"write_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":2},"status":{"status":"write"}}},{"id":"1_1594620996.035328_199.1","section":"bucket.instance","name":"cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25","timestamp":"2020-07-13T06:16:36.035328199Z","data":{"read_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":1},"write_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":2},"status":{"status":"complete"}}},{"id":"1_1594620996.042162_200.1","section":"bucket.instance","name":"cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26","timestamp":"2020-07-13T06:16:36.042162134Z","data":{"read_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":1},"write_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":2},"status":{"status":"write"}}},{"id":"1_1594620996.046721_201.1","section":"bucket.instance","name":"cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26","timestamp":"2020-07-13T06:16:36.046721459Z","data":{"read_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":1},"write_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":2},"status":{"status":"complete"}}},{"id":"1_1594620996.060062_202.1","section":"bucket.instance","name":"cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27","timestamp":"2020-07-13T06:16:36.060062705Z","data":{"read_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":1},"write_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":2},"status":{"status":"write"}}},{"id":"1_1594620996.066494_203.1","section":"bucket.instance","name":"cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27","timestamp":"2020-07-13T06:16:36.066494340Z","data":{"read_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":1},"write_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621170.096466_265.1","section":"bucket.instance","name":"cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28","timestamp":"2020-07-13T06:19:30.096466874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":1},"status":{"status":"write"}}},{"id":"1_1594621170.101095_286.1","section":"bucket.instance","name":"cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28","timestamp":"2020-07-13T06:19:30.101095792Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621170.103270_287.1","section":"bucket","name":"cngrzz-19","timestamp":"2020-07-13T06:19:30.103270825Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wQI4zfCNmpgzGLmtq7JYIcF","ver":1},"status":{"status":"write"}}},{"id":"1_1594621170.107053_288.1","section":"bucket","name":"cngrzz-19","timestamp":"2020-07-13T06:19:30.107053912Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wQI4zfCNmpgzGLmtq7JYIcF","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621170.117756_289.1","section":"bucket.instance","name":"cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28","timestamp":"2020-07-13T06:19:30.117756676Z","data":{"read_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":1},"write_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":2},"status":{"status":"write"}}},{"id":"1_1594621170.123767_290.1","section":"bucket.instance","name":"cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28","timestamp":"2020-07-13T06:19:30.123767219Z","data":{"read_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":1},"write_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621322.052070_335.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:22:02.052070538Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":1},"status":{"status":"write"}}},{"id":"1_1594621322.057256_350.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:22:02.057256567Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621322.059913_351.1","section":"bucket","name":"cngrzz-20","timestamp":"2020-07-13T06:22:02.059913372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SiUnqH4drkKjEAXU4lM6KVi","ver":1},"status":{"status":"write"}}},{"id":"1_1594621322.065336_352.1","section":"bucket","name":"cngrzz-20","timestamp":"2020-07-13T06:22:02.065336451Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SiUnqH4drkKjEAXU4lM6KVi","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621329.476722_365.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:22:09.476722756Z","data":{"read_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":1},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":2},"status":{"status":"write"}}},{"id":"1_1594621329.487977_368.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:22:09.487977095Z","data":{"read_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":1},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621390.911893_402.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:23:10.911893325Z","data":{"read_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":2},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":3},"status":{"status":"write"}}},{"id":"1_1594621390.921870_405.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:23:10.921870742Z","data":{"read_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":2},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":3},"status":{"status":"complete"}}},{"id":"1_1594621433.317645_430.1","section":"bucket.instance","name":"cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30","timestamp":"2020-07-13T06:23:53.317645856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":1},"status":{"status":"write"}}},{"id":"1_1594621433.322934_433.1","section":"bucket.instance","name":"cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30","timestamp":"2020-07-13T06:23:53.322934869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621433.324814_434.1","section":"bucket","name":"cngrzz-21","timestamp":"2020-07-13T06:23:53.324814398Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vqw1RgtzigBqWBzMKd9qNek","ver":1},"status":{"status":"write"}}},{"id":"1_1594621433.328408_435.1","section":"bucket","name":"cngrzz-21","timestamp":"2020-07-13T06:23:53.328408882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vqw1RgtzigBqWBzMKd9qNek","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621433.386628_436.1","section":"bucket.instance","name":"cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31","timestamp":"2020-07-13T06:23:53.386628999Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":1},"status":{"status":"write"}}},{"id":"1_1594621433.390873_437.1","section":"bucket.instance","name":"cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31","timestamp":"2020-07-13T06:23:53.390873686Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621433.393237_438.1","section":"bucket","name":"cngrzz-22","timestamp":"2020-07-13T06:23:53.393237663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7bwonnuvmzNIIPT8mR6P0bA","ver":1},"status":{"status":"write"}}},{"id":"1_1594621433.397692_439.1","section":"bucket","name":"cngrzz-22","timestamp":"2020-07-13T06:23:53.397692663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7bwonnuvmzNIIPT8mR6P0bA","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621434.232311_440.1","section":"bucket.instance","name":"cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32","timestamp":"2020-07-13T06:23:54.232311850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":1},"status":{"status":"write"}}},{"id":"1_1594621434.236794_441.1","section":"bucket.instance","name":"cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32","timestamp":"2020-07-13T06:23:54.236794211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621434.238800_442.1","section":"bucket","name":"cngrzz-23","timestamp":"2020-07-13T06:23:54.238800428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SvlhobBY9Slsbwne1CTEC-L","ver":1},"status":{"status":"write"}}},{"id":"1_1594621434.242788_443.1","section":"bucket","name":"cngrzz-23","timestamp":"2020-07-13T06:23:54.242788736Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SvlhobBY9Slsbwne1CTEC-L","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621434.264342_444.1","section":"bucket.instance","name":"cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30","timestamp":"2020-07-13T06:23:54.264342494Z","data":{"read_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":1},"write_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":2},"status":{"status":"write"}}},{"id":"1_1594621434.270547_445.1","section":"bucket.instance","name":"cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30","timestamp":"2020-07-13T06:23:54.270547881Z","data":{"read_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":1},"write_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621434.275368_446.1","section":"bucket.instance","name":"cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31","timestamp":"2020-07-13T06:23:54.275368951Z","data":{"read_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":1},"write_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":2},"status":{"status":"write"}}},{"id":"1_1594621434.282613_447.1","section":"bucket.instance","name":"cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31","timestamp":"2020-07-13T06:23:54.282613618Z","data":{"read_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":1},"write_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621434.294881_448.1","section":"bucket.instance","name":"cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32","timestamp":"2020-07-13T06:23:54.294881064Z","data":{"read_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":1},"write_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":2},"status":{"status":"write"}}},{"id":"1_1594621434.303397_449.1","section":"bucket.instance","name":"cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32","timestamp":"2020-07-13T06:23:54.303397214Z","data":{"read_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":1},"write_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621455.307113_481.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.307113064Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.311789_492.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.311789146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.314390_493.1","section":"bucket","name":"cngrzz-24","timestamp":"2020-07-13T06:24:15.314390695Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_syYOTmiVtbmFjOCG8p3NuzV","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.318379_494.1","section":"bucket","name":"cngrzz-24","timestamp":"2020-07-13T06:24:15.318379915Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_syYOTmiVtbmFjOCG8p3NuzV","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.344614_495.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.344614344Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.349148_496.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.349148544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.351560_497.1","section":"bucket","name":"cngrzz-25","timestamp":"2020-07-13T06:24:15.351560923Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L6eQ90-G-twMkpkPL-m5q_J","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.355886_498.1","section":"bucket","name":"cngrzz-25","timestamp":"2020-07-13T06:24:15.355886460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L6eQ90-G-twMkpkPL-m5q_J","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.405945_499.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.405945564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.410863_500.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.410863468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.412859_501.1","section":"bucket","name":"cngrzz-26","timestamp":"2020-07-13T06:24:15.412859756Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wVuwzieS4x5WATempXhxurc","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.417215_502.1","section":"bucket","name":"cngrzz-26","timestamp":"2020-07-13T06:24:15.417215461Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wVuwzieS4x5WATempXhxurc","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.441624_503.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.441624712Z","data":{"read_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":1},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":2},"status":{"status":"write"}}},{"id":"1_1594621455.464024_504.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.464024694Z","data":{"read_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":1},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621455.470869_505.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.470869735Z","data":{"read_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":2},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":3},"status":{"status":"write"}}},{"id":"1_1594621455.475511_506.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.475511044Z","data":{"read_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":2},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":3},"status":{"status":"complete"}}},{"id":"1_1594621455.479588_507.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.479588113Z","data":{"read_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":1},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":2},"status":{"status":"write"}}},{"id":"1_1594621455.484136_508.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.484136677Z","data":{"read_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":1},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621455.494806_509.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.494806032Z","data":{"read_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":2},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":3},"status":{"status":"write"}}},{"id":"1_1594621455.505325_510.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.505325897Z","data":{"read_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":2},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":3},"status":{"status":"complete"}}},{"id":"1_1594621455.518131_511.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.518131893Z","data":{"read_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":1},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":2},"status":{"status":"write"}}},{"id":"1_1594621455.527513_512.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.527513994Z","data":{"read_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":1},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621455.542157_513.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.542157223Z","data":{"read_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":2},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":3},"status":{"status":"write"}}},{"id":"1_1594621455.551984_514.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.551984028Z","data":{"read_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":2},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":3},"status":{"status":"complete"}}},{"id":"1_1594621478.463758_549.1","section":"bucket.instance","name":"cngrzz-27:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.36","timestamp":"2020-07-13T06:24:38.463758191Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.468096_556.1","section":"bucket.instance","name":"cngrzz-27:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.36","timestamp":"2020-07-13T06:24:38.468096388Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.470142_557.1","section":"bucket","name":"cngrzz-27","timestamp":"2020-07-13T06:24:38.470142675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xKmuAPoskvLbXPU03OlC73s","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.473856_558.1","section":"bucket","name":"cngrzz-27","timestamp":"2020-07-13T06:24:38.473856655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xKmuAPoskvLbXPU03OlC73s","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.497896_559.1","section":"bucket.instance","name":"cngrzz-28:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.37","timestamp":"2020-07-13T06:24:38.497896309Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.502362_560.1","section":"bucket.instance","name":"cngrzz-28:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.37","timestamp":"2020-07-13T06:24:38.502362092Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.504706_561.1","section":"bucket","name":"cngrzz-28","timestamp":"2020-07-13T06:24:38.504706153Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m0vuxwdxcV4K0zwl3eB9Wa9","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.509299_562.1","section":"bucket","name":"cngrzz-28","timestamp":"2020-07-13T06:24:38.509299001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m0vuxwdxcV4K0zwl3eB9Wa9","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.554287_563.1","section":"bucket.instance","name":"cngrzz-29:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.38","timestamp":"2020-07-13T06:24:38.554287987Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.561718_564.1","section":"bucket.instance","name":"cngrzz-29:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.38","timestamp":"2020-07-13T06:24:38.561718878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.564680_565.1","section":"bucket","name":"cngrzz-29","timestamp":"2020-07-13T06:24:38.564680554Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2TuYKigZrzuv-dMOyxValfc","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.569842_566.1","section":"bucket","name":"cngrzz-29","timestamp":"2020-07-13T06:24:38.569842326Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2TuYKigZrzuv-dMOyxValfc","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.595037_567.1","section":"bucket.instance","name":"cngrzz-27:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.36","timestamp":"2020-07-13T06:24:38.595037740Z","data":{"read_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":1},"write_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":2},"status":{"status":"write"}}},{"id":"1_1594621478.600762_568.1","section":"bucket.instance","name":"cngrzz-27:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.36","timestamp":"2020-07-13T06:24:38.600762996Z","data":{"read_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":1},"write_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621478.647960_569.1","section":"bucket.instance","name":"cngrzz-28:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.37","timestamp":"2020-07-13T06:24:38.647960510Z","data":{"read_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":1},"write_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":2},"status":{"status":"write"}}},{"id":"1_1594621478.658940_570.1","section":"bucket.instance","name":"cngrzz-28:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.37","timestamp":"2020-07-13T06:24:38.658940233Z","data":{"read_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":1},"write_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621478.723174_571.1","section":"bucket.instance","name":"cngrzz-29:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.38","timestamp":"2020-07-13T06:24:38.723174021Z","data":{"read_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":1},"write_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":2},"status":{"status":"write"}}},{"id":"1_1594621478.729364_572.1","section":"bucket.instance","name":"cngrzz-29:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.38","timestamp":"2020-07-13T06:24:38.729364605Z","data":{"read_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":1},"write_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621478.803680_573.1","section":"bucket.instance","name":"cngrzz-30:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.39","timestamp":"2020-07-13T06:24:38.803680169Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.808550_574.1","section":"bucket.instance","name":"cngrzz-30:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.39","timestamp":"2020-07-13T06:24:38.808550868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.810788_575.1","section":"bucket","name":"cngrzz-30","timestamp":"2020-07-13T06:24:38.810788109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tcbDPAiLUbg1Qosf4aVJZaR","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.814964_576.1","section":"bucket","name":"cngrzz-30","timestamp":"2020-07-13T06:24:38.814964209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tcbDPAiLUbg1Qosf4aVJZaR","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.842441_577.1","section":"bucket.instance","name":"cngrzz-31:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.40","timestamp":"2020-07-13T06:24:38.842441698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.846740_578.1","section":"bucket.instance","name":"cngrzz-31:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.40","timestamp":"2020-07-13T06:24:38.846740063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.849242_579.1","section":"bucket","name":"cngrzz-31","timestamp":"2020-07-13T06:24:38.849242603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PcNMXYzKJiqzUL9v_g9UUEW","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.853963_580.1","section":"bucket","name":"cngrzz-31","timestamp":"2020-07-13T06:24:38.853963184Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PcNMXYzKJiqzUL9v_g9UUEW","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.938346_581.1","section":"bucket.instance","name":"cngrzz-32:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.41","timestamp":"2020-07-13T06:24:38.938346254Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.942593_582.1","section":"bucket.instance","name":"cngrzz-32:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.41","timestamp":"2020-07-13T06:24:38.942593554Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.944644_583.1","section":"bucket","name":"cngrzz-32","timestamp":"2020-07-13T06:24:38.944644272Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cgRpl2sxiftnVUa2OO_gEgo","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.948537_584.1","section":"bucket","name":"cngrzz-32","timestamp":"2020-07-13T06:24:38.948537812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cgRpl2sxiftnVUa2OO_gEgo","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.974066_585.1","section":"bucket.instance","name":"cngrzz-30:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.39","timestamp":"2020-07-13T06:24:38.974066414Z","data":{"read_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":1},"write_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":2},"status":{"status":"write"}}},{"id":"1_1594621478.980780_586.1","section":"bucket.instance","name":"cngrzz-30:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.39","timestamp":"2020-07-13T06:24:38.980780610Z","data":{"read_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":1},"write_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621479.033952_587.1","section":"bucket.instance","name":"cngrzz-31:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.40","timestamp":"2020-07-13T06:24:39.033952236Z","data":{"read_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":1},"write_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":2},"status":{"status":"write"}}},{"id":"1_1594621479.045331_588.1","section":"bucket.instance","name":"cngrzz-31:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.40","timestamp":"2020-07-13T06:24:39.045331654Z","data":{"read_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":1},"write_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621479.109134_589.1","section":"bucket.instance","name":"cngrzz-32:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.41","timestamp":"2020-07-13T06:24:39.109134981Z","data":{"read_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":1},"write_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":2},"status":{"status":"write"}}},{"id":"1_1594621479.118646_590.1","section":"bucket.instance","name":"cngrzz-32:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.41","timestamp":"2020-07-13T06:24:39.118646708Z","data":{"read_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":1},"write_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621479.194094_591.1","section":"bucket.instance","name":"cngrzz-33:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.42","timestamp":"2020-07-13T06:24:39.194094933Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.200722_592.1","section":"bucket.instance","name":"cngrzz-33:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.42","timestamp":"2020-07-13T06:24:39.200722310Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.203027_593.1","section":"bucket","name":"cngrzz-33","timestamp":"2020-07-13T06:24:39.203027753Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PuawPXsI3tXqBy_MXXvr0Cy","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.209920_594.1","section":"bucket","name":"cngrzz-33","timestamp":"2020-07-13T06:24:39.209920207Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PuawPXsI3tXqBy_MXXvr0Cy","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.241265_595.1","section":"bucket.instance","name":"cngrzz-34:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.43","timestamp":"2020-07-13T06:24:39.241265494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.246173_596.1","section":"bucket.instance","name":"cngrzz-34:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.43","timestamp":"2020-07-13T06:24:39.246173690Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.248646_597.1","section":"bucket","name":"cngrzz-34","timestamp":"2020-07-13T06:24:39.248646104Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SZllY3UvM6YWJqOBtYiiacb","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.253617_598.1","section":"bucket","name":"cngrzz-34","timestamp":"2020-07-13T06:24:39.253617145Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SZllY3UvM6YWJqOBtYiiacb","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.323567_599.1","section":"bucket.instance","name":"cngrzz-35:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.44","timestamp":"2020-07-13T06:24:39.323567314Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.328276_600.1","section":"bucket.instance","name":"cngrzz-35:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.44","timestamp":"2020-07-13T06:24:39.328276870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.330806_601.1","section":"bucket","name":"cngrzz-35","timestamp":"2020-07-13T06:24:39.330806173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v-R90_jF6GVW73YIo8cHfq0","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.336092_602.1","section":"bucket","name":"cngrzz-35","timestamp":"2020-07-13T06:24:39.336092549Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v-R90_jF6GVW73YIo8cHfq0","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.359801_603.1","section":"bucket.instance","name":"cngrzz-33:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.42","timestamp":"2020-07-13T06:24:39.359801153Z","data":{"read_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":1},"write_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":2},"status":{"status":"write"}}},{"id":"1_1594621479.369370_604.1","section":"bucket.instance","name":"cngrzz-33:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.42","timestamp":"2020-07-13T06:24:39.369370295Z","data":{"read_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":1},"write_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621480.420657_605.1","section":"bucket.instance","name":"cngrzz-34:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.43","timestamp":"2020-07-13T06:24:40.420657898Z","data":{"read_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":1},"write_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":2},"status":{"status":"write"}}},{"id":"1_1594621480.432601_606.1","section":"bucket.instance","name":"cngrzz-34:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.43","timestamp":"2020-07-13T06:24:40.432601102Z","data":{"read_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":1},"write_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621482.474186_607.1","section":"bucket.instance","name":"cngrzz-35:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.44","timestamp":"2020-07-13T06:24:42.474186915Z","data":{"read_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":1},"write_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":2},"status":{"status":"write"}}},{"id":"1_1594621482.484822_608.1","section":"bucket.instance","name":"cngrzz-35:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.44","timestamp":"2020-07-13T06:24:42.484822401Z","data":{"read_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":1},"write_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621482.913401_609.1","section":"bucket.instance","name":"cngrzz-36:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.45","timestamp":"2020-07-13T06:24:42.913401257Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BHYIPGqLECVNEi2cZ-MYjOz","ver":1},"status":{"status":"write"}}},{"id":"1_1594621482.917800_610.1","section":"bucket.instance","name":"cngrzz-36:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.45","timestamp":"2020-07-13T06:24:42.917800056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BHYIPGqLECVNEi2cZ-MYjOz","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621482.920043_611.1","section":"bucket","name":"cngrzz-36","timestamp":"2020-07-13T06:24:42.920043693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lDf1FKkxnRubfsQ9M1--dgD","ver":1},"status":{"status":"write"}}},{"id":"1_1594621482.924506_612.1","section":"bucket","name":"cngrzz-36","timestamp":"2020-07-13T06:24:42.924506470Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lDf1FKkxnRubfsQ9M1--dgD","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621483.015114_613.1","section":"bucket.instance","name":"cngrzz-37:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.46","timestamp":"2020-07-13T06:24:43.015114675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oP_SUc8PxL6c4XMdnGrcIDd","ver":1},"status":{"status":"write"}}},{"id":"1_1594621483.020538_614.1","section":"bucket.instance","name":"cngrzz-37:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.46","timestamp":"2020-07-13T06:24:43.020538172Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oP_SUc8PxL6c4XMdnGrcIDd","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621483.022545_615.1","section":"bucket","name":"cngrzz-37","timestamp":"2020-07-13T06:24:43.022545950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5gFmRJU_Lnfca_c3ktRTso7","ver":1},"status":{"status":"write"}}},{"id":"1_1594621483.026371_616.1","section":"bucket","name":"cngrzz-37","timestamp":"2020-07-13T06:24:43.026371722Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5gFmRJU_Lnfca_c3ktRTso7","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621483.081477_617.1","section":"bucket.instance","name":"cngrzz-38:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.47","timestamp":"2020-07-13T06:24:43.081477070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nVm3w2ql0Kn2oqc2UsEJH8a","ver":1},"status":{"status":"write"}}},{"id":"1_1594621483.087242_618.1","section":"bucket.instance","name":"cngrzz-38:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.47","timestamp":"2020-07-13T06:24:43.087242473Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nVm3w2ql0Kn2oqc2UsEJH8a","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621483.089407_619.1","section":"bucket","name":"cngrzz-38","timestamp":"2020-07-13T06:24:43.089407632Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3eyVOL0MMcQBz9oFqM87zxz","ver":1},"status":{"status":"write"}}},{"id":"1_1594621483.093117_620.1","section":"bucket","name":"cngrzz-38","timestamp":"2020-07-13T06:24:43.093117980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3eyVOL0MMcQBz9oFqM87zxz","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.461368_621.1","section":"bucket.instance","name":"cngrzz-39:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.48","timestamp":"2020-07-13T06:24:46.461368112Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uMnm_CbH26xOW2j2VokpY4s","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.466851_624.1","section":"bucket.instance","name":"cngrzz-39:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.48","timestamp":"2020-07-13T06:24:46.466851306Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uMnm_CbH26xOW2j2VokpY4s","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.469013_625.1","section":"bucket","name":"cngrzz-39","timestamp":"2020-07-13T06:24:46.469013868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eqEaW0o9d4QYozEF28u7c49","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.472599_626.1","section":"bucket","name":"cngrzz-39","timestamp":"2020-07-13T06:24:46.472599369Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eqEaW0o9d4QYozEF28u7c49","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.759727_627.1","section":"bucket.instance","name":"cngrzz-40:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.49","timestamp":"2020-07-13T06:24:46.759727960Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OP3txqbvev88w0dP9vK-dO5","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.765631_628.1","section":"bucket.instance","name":"cngrzz-40:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.49","timestamp":"2020-07-13T06:24:46.765631411Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OP3txqbvev88w0dP9vK-dO5","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.767996_629.1","section":"bucket","name":"cngrzz-40","timestamp":"2020-07-13T06:24:46.767996150Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M3LKRjOIblMfECEncxsHzyu","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.772433_630.1","section":"bucket","name":"cngrzz-40","timestamp":"2020-07-13T06:24:46.772433975Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M3LKRjOIblMfECEncxsHzyu","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.817605_631.1","section":"bucket.instance","name":"cngrzz-41:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.50","timestamp":"2020-07-13T06:24:46.817605518Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kHTWaobdOcV3WvsIXS-4JDw","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.823059_632.1","section":"bucket.instance","name":"cngrzz-41:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.50","timestamp":"2020-07-13T06:24:46.823059818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kHTWaobdOcV3WvsIXS-4JDw","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.825455_633.1","section":"bucket","name":"cngrzz-41","timestamp":"2020-07-13T06:24:46.825455311Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_enzpzPLLpAzWWe8NZI6ZjKh","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.829707_634.1","section":"bucket","name":"cngrzz-41","timestamp":"2020-07-13T06:24:46.829707023Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_enzpzPLLpAzWWe8NZI6ZjKh","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620914.565843_67.1","section":"bucket.instance","name":"cngrzz-1:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.1","timestamp":"2020-07-13T06:15:14.565843290Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NrX7NUBlKlctPxLgZQhVZeJ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620914.569624_68.1","section":"bucket.instance","name":"cngrzz-1:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.1","timestamp":"2020-07-13T06:15:14.569624562Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NrX7NUBlKlctPxLgZQhVZeJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620914.571621_69.1","section":"bucket","name":"cngrzz-1","timestamp":"2020-07-13T06:15:14.571621151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B_xEGfjqp5GJf-kckzrymxy","ver":1},"status":{"status":"write"}}},{"id":"1_1594620914.576799_70.1","section":"bucket","name":"cngrzz-1","timestamp":"2020-07-13T06:15:14.576799306Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B_xEGfjqp5GJf-kckzrymxy","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620914.604902_71.1","section":"bucket.instance","name":"cngrzz-2:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.2","timestamp":"2020-07-13T06:15:14.604902219Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PvZsoXHBEggBhfEef7o5U0T","ver":1},"status":{"status":"write"}}},{"id":"1_1594620914.609491_72.1","section":"bucket.instance","name":"cngrzz-2:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.2","timestamp":"2020-07-13T06:15:14.609491570Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PvZsoXHBEggBhfEef7o5U0T","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620914.612054_73.1","section":"bucket","name":"cngrzz-2","timestamp":"2020-07-13T06:15:14.612054698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6IT0MMiYAzd8ADA6mJQGl2V","ver":1},"status":{"status":"write"}}},{"id":"1_1594620914.620627_74.1","section":"bucket","name":"cngrzz-2","timestamp":"2020-07-13T06:15:14.620627832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6IT0MMiYAzd8ADA6mJQGl2V","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620918.616085_75.1","section":"bucket.instance","name":"cngrzz-3:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.3","timestamp":"2020-07-13T06:15:18.616085899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jNBNF8gIfCQ0VtU1Nd6VsXo","ver":1},"status":{"status":"write"}}},{"id":"1_1594620918.620674_76.1","section":"bucket.instance","name":"cngrzz-3:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.3","timestamp":"2020-07-13T06:15:18.620674736Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jNBNF8gIfCQ0VtU1Nd6VsXo","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620918.625187_77.1","section":"bucket","name":"cngrzz-3","timestamp":"2020-07-13T06:15:18.625187717Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6UxFHip67VoBBxh6FLU5ArZ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620918.629897_78.1","section":"bucket","name":"cngrzz-3","timestamp":"2020-07-13T06:15:18.629897499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6UxFHip67VoBBxh6FLU5ArZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.868689_79.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.4","timestamp":"2020-07-13T06:15:21.868689028Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5eRMS50W9omalXkI-eKEkZH","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.873048_80.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.4","timestamp":"2020-07-13T06:15:21.873048233Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5eRMS50W9omalXkI-eKEkZH","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.874813_81.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:21.874813555Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JboyausbemE9wke0D-OFRJL","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.881925_82.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:21.881925098Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JboyausbemE9wke0D-OFRJL","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.904963_83.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.5","timestamp":"2020-07-13T06:15:21.904963978Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_orhbCdPThQDntboRt2h0Sl9","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.910649_84.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.5","timestamp":"2020-07-13T06:15:21.910649089Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_orhbCdPThQDntboRt2h0Sl9","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.913196_85.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:21.913196616Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W9AKCYVnn-UeTSsMsys7MT_","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.920774_86.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:21.920774888Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W9AKCYVnn-UeTSsMsys7MT_","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.970374_87.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.6","timestamp":"2020-07-13T06:15:21.970374493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v6ZB9vPFQIwn4oxtloegK6s","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.976019_88.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.6","timestamp":"2020-07-13T06:15:21.976019891Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v6ZB9vPFQIwn4oxtloegK6s","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.980716_89.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:21.980716496Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IjM-H8ZOw8IgEVRDW0EGVzm","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.991312_90.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:21.991312189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IjM-H8ZOw8IgEVRDW0EGVzm","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.389888_91.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.7","timestamp":"2020-07-13T06:15:25.389888371Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RWdhBH265JDcuhXxt8yJvwV","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.394176_92.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.7","timestamp":"2020-07-13T06:15:25.394176539Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RWdhBH265JDcuhXxt8yJvwV","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.396386_93.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:25.396386821Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t0iFgy3ChisA7Hu7SEB54i8","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.400663_94.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:25.400663561Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t0iFgy3ChisA7Hu7SEB54i8","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.431105_95.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.8","timestamp":"2020-07-13T06:15:25.431105188Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OXxKeva7ZQcmEiQXaZBhcee","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.435395_96.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.8","timestamp":"2020-07-13T06:15:25.435395684Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OXxKeva7ZQcmEiQXaZBhcee","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.440261_97.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:25.440261819Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IbupanJOb1vHvGDroO9OyV1","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.444849_98.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:25.444849981Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IbupanJOb1vHvGDroO9OyV1","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.475021_99.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.9","timestamp":"2020-07-13T06:15:25.475021111Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sws_fFW3wnseWgGw3cH1pX-","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.479177_100.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.9","timestamp":"2020-07-13T06:15:25.479177003Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sws_fFW3wnseWgGw3cH1pX-","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.481306_101.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:25.481306318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Lq9uI5UBhcdqocC2jNTLpQ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.486249_102.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:25.486249916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Lq9uI5UBhcdqocC2jNTLpQ","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.679609_103.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.10","timestamp":"2020-07-13T06:15:25.679609508Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mlzW7asaUOM35pXKhvz5VX2","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.684249_104.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.10","timestamp":"2020-07-13T06:15:25.684249287Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mlzW7asaUOM35pXKhvz5VX2","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.686238_105.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:25.686238812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IRKdAJCiQhoxy_hbM2Zp4rZ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.690770_106.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:25.690770954Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IRKdAJCiQhoxy_hbM2Zp4rZ","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.750744_107.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.11","timestamp":"2020-07-13T06:15:25.750744600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TVxzlNwS0AToOujH53HcOeO","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.755250_108.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.11","timestamp":"2020-07-13T06:15:25.755250926Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TVxzlNwS0AToOujH53HcOeO","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.759191_109.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:25.759191648Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__V6H1e1TkzH83WDQjMnLs3J","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.763788_110.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:25.763788507Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__V6H1e1TkzH83WDQjMnLs3J","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.874919_111.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.12","timestamp":"2020-07-13T06:15:25.874919330Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cppER_j32QTT6jfkUc7OtN9","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.880056_112.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.12","timestamp":"2020-07-13T06:15:25.880056550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cppER_j32QTT6jfkUc7OtN9","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.882580_113.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:25.882580782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1kbjCgsQlkaQET_-cKcofT8","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.887578_114.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:25.887578813Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1kbjCgsQlkaQET_-cKcofT8","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620926.372198_115.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.13","timestamp":"2020-07-13T06:15:26.372198300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AbGEn19L1mso3vjlMICZuzH","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.376648_116.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.13","timestamp":"2020-07-13T06:15:26.376648564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AbGEn19L1mso3vjlMICZuzH","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620926.379137_117.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:26.379137187Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AJ1-E53HupEVn1PaRJLxknN","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.385397_118.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:26.385397048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AJ1-E53HupEVn1PaRJLxknN","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620926.440877_119.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.14","timestamp":"2020-07-13T06:15:26.440877932Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QwRkzlppS0ZPFANJbrxRLAo","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.445773_120.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.14","timestamp":"2020-07-13T06:15:26.445773115Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QwRkzlppS0ZPFANJbrxRLAo","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620926.447974_121.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:26.447974475Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sXT0pkowqKazgGp3FXuQLsA","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.452307_122.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:26.452307296Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sXT0pkowqKazgGp3FXuQLsA","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620926.558326_123.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.15","timestamp":"2020-07-13T06:15:26.558326355Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OrXmOtQEOSJJK03Tf2okhoA","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.563257_124.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.15","timestamp":"2020-07-13T06:15:26.563257650Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OrXmOtQEOSJJK03Tf2okhoA","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620926.565740_125.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:26.565740914Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AF1yTvVufSmQzFmQD6m6PPG","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.570739_126.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:26.570739160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AF1yTvVufSmQzFmQD6m6PPG","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620929.001475_127.1","section":"bucket.instance","name":"cngrzz-7:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.16","timestamp":"2020-07-13T06:15:29.001475256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ng8rN4G68wRQ3eDQZ46UDUy","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.006047_128.1","section":"bucket.instance","name":"cngrzz-7:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.16","timestamp":"2020-07-13T06:15:29.006047645Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ng8rN4G68wRQ3eDQZ46UDUy","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.008050_129.1","section":"bucket","name":"cngrzz-7","timestamp":"2020-07-13T06:15:29.008050687Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.012811_130.1","section":"bucket","name":"cngrzz-7","timestamp":"2020-07-13T06:15:29.012811406Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.036306_131.1","section":"bucket.instance","name":"cngrzz-8:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.17","timestamp":"2020-07-13T06:15:29.036306449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AreU9GaZl3bJuU19nqk4OSY","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.040693_132.1","section":"bucket.instance","name":"cngrzz-8:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.17","timestamp":"2020-07-13T06:15:29.040693341Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AreU9GaZl3bJuU19nqk4OSY","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.042995_133.1","section":"bucket","name":"cngrzz-8","timestamp":"2020-07-13T06:15:29.042995034Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.047392_134.1","section":"bucket","name":"cngrzz-8","timestamp":"2020-07-13T06:15:29.047392137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.090400_135.1","section":"bucket.instance","name":"cngrzz-9:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.18","timestamp":"2020-07-13T06:15:29.090400277Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k8n1zW2nPOQWnZG2f7x7kJc","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.094794_136.1","section":"bucket.instance","name":"cngrzz-9:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.18","timestamp":"2020-07-13T06:15:29.094794643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k8n1zW2nPOQWnZG2f7x7kJc","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.096824_137.1","section":"bucket","name":"cngrzz-9","timestamp":"2020-07-13T06:15:29.096824917Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.100492_138.1","section":"bucket","name":"cngrzz-9","timestamp":"2020-07-13T06:15:29.100492014Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620932.680034_139.1","section":"bucket","name":"cngrzz-7","timestamp":"2020-07-13T06:15:32.680034714Z","data":{"read_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":1},"write_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":2},"status":{"status":"remove"}}},{"id":"1_1594620932.686075_140.1","section":"bucket","name":"cngrzz-7","timestamp":"2020-07-13T06:15:32.686075353Z","data":{"read_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":1},"write_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":2},"status":{"status":"complete"}}},{"id":"1_1594620933.010327_141.1","section":"bucket","name":"cngrzz-8","timestamp":"2020-07-13T06:15:33.010327021Z","data":{"read_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":1},"write_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":2},"status":{"status":"remove"}}},{"id":"1_1594620933.017231_142.1","section":"bucket","name":"cngrzz-8","timestamp":"2020-07-13T06:15:33.017231115Z","data":{"read_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":1},"write_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":2},"status":{"status":"complete"}}},{"id":"1_1594620933.436438_143.1","section":"bucket","name":"cngrzz-9","timestamp":"2020-07-13T06:15:33.436438596Z","data":{"read_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":1},"write_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":2},"status":{"status":"remove"}}},{"id":"1_1594620933.443793_144.1","section":"bucket","name":"cngrzz-9","timestamp":"2020-07-13T06:15:33.443793026Z","data":{"read_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":1},"write_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":2},"status":{"status":"complete"}}}] 2020-07-13T06:27:06.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '587bd98b-69d2-49ba-9dbd-2be72177aade', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:06.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '587bd98b-69d2-49ba-9dbd-2be72177aade', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:06.058 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 587bd98b-69d2-49ba-9dbd-2be72177aade --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:06.098 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:06.098 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:06.355 INFO:teuthology.orchestra.run.smithi014.stdout:[{"id":"1_1594620792.560320_57.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-07-13T06:13:12.560320432Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w5Jia4Y6RKNUd9E9jGppGgY","ver":1},"status":{"status":"write"}}},{"id":"1_1594620792.569571_63.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-07-13T06:13:12.569571770Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w5Jia4Y6RKNUd9E9jGppGgY","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.353592_71.1","section":"bucket.instance","name":"cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19","timestamp":"2020-07-13T06:15:35.353592485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5YWV65SHwhZo1_hTFsOthdm","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.357969_84.1","section":"bucket.instance","name":"cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19","timestamp":"2020-07-13T06:15:35.357969060Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5YWV65SHwhZo1_hTFsOthdm","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.360453_85.1","section":"bucket","name":"cngrzz-10","timestamp":"2020-07-13T06:15:35.360453117Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lyi7_9F-fQavvS6ttQb4QqQ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.365063_86.1","section":"bucket","name":"cngrzz-10","timestamp":"2020-07-13T06:15:35.365063336Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lyi7_9F-fQavvS6ttQb4QqQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.387630_87.1","section":"bucket.instance","name":"cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20","timestamp":"2020-07-13T06:15:35.387630766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XJOAFF_eewRAyz7Ek55C2eZ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.393964_88.1","section":"bucket.instance","name":"cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20","timestamp":"2020-07-13T06:15:35.393964283Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XJOAFF_eewRAyz7Ek55C2eZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.396127_89.1","section":"bucket","name":"cngrzz-11","timestamp":"2020-07-13T06:15:35.396127319Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uYUbUU7eUol0NqPi_EWhoBS","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.401731_90.1","section":"bucket","name":"cngrzz-11","timestamp":"2020-07-13T06:15:35.401731762Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uYUbUU7eUol0NqPi_EWhoBS","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.439458_91.1","section":"bucket.instance","name":"cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21","timestamp":"2020-07-13T06:15:35.439458898Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y5EI8u7QgEs_npG_hSdZdhk","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.447280_92.1","section":"bucket.instance","name":"cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21","timestamp":"2020-07-13T06:15:35.447280663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y5EI8u7QgEs_npG_hSdZdhk","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620935.451453_93.1","section":"bucket","name":"cngrzz-12","timestamp":"2020-07-13T06:15:35.451453816Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1xVfXDesczYGyPwaO94C87T","ver":1},"status":{"status":"write"}}},{"id":"1_1594620935.457356_94.1","section":"bucket","name":"cngrzz-12","timestamp":"2020-07-13T06:15:35.457356063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1xVfXDesczYGyPwaO94C87T","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.556299_103.1","section":"bucket.instance","name":"cngrzz-13:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.22","timestamp":"2020-07-13T06:15:53.556299812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqQn6_rjhfsJuMkDSydnu0U","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.563929_114.1","section":"bucket.instance","name":"cngrzz-13:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.22","timestamp":"2020-07-13T06:15:53.563929704Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqQn6_rjhfsJuMkDSydnu0U","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.567882_115.1","section":"bucket","name":"cngrzz-13","timestamp":"2020-07-13T06:15:53.567882599Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_raNSztBe-9dO0DB5H_S-No6","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.575383_116.1","section":"bucket","name":"cngrzz-13","timestamp":"2020-07-13T06:15:53.575383568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_raNSztBe-9dO0DB5H_S-No6","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.610132_118.1","section":"bucket.instance","name":"cngrzz-14:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.23","timestamp":"2020-07-13T06:15:53.610132033Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uUpk1HTypA7roPDy4mLwqnM","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.614384_119.1","section":"bucket.instance","name":"cngrzz-14:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.23","timestamp":"2020-07-13T06:15:53.614384828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uUpk1HTypA7roPDy4mLwqnM","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.616654_120.1","section":"bucket","name":"cngrzz-14","timestamp":"2020-07-13T06:15:53.616654760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9zhZRNYkovqDEm3EDgLOk8P","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.622281_121.1","section":"bucket","name":"cngrzz-14","timestamp":"2020-07-13T06:15:53.622281110Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9zhZRNYkovqDEm3EDgLOk8P","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.669597_122.1","section":"bucket.instance","name":"cngrzz-15:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.24","timestamp":"2020-07-13T06:15:53.669597829Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_80eR23_oL7sRegm5pYbqCb3","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.675879_123.1","section":"bucket.instance","name":"cngrzz-15:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.24","timestamp":"2020-07-13T06:15:53.675879396Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_80eR23_oL7sRegm5pYbqCb3","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620953.677770_124.1","section":"bucket","name":"cngrzz-15","timestamp":"2020-07-13T06:15:53.677770571Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0h_SR2fNa6Li7p9B9ibEN6V","ver":1},"status":{"status":"write"}}},{"id":"1_1594620953.682997_125.1","section":"bucket","name":"cngrzz-15","timestamp":"2020-07-13T06:15:53.682997873Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0h_SR2fNa6Li7p9B9ibEN6V","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.903801_180.1","section":"bucket.instance","name":"cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25","timestamp":"2020-07-13T06:16:35.903801109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.908369_187.1","section":"bucket.instance","name":"cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25","timestamp":"2020-07-13T06:16:35.908369870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.910031_188.1","section":"bucket","name":"cngrzz-16","timestamp":"2020-07-13T06:16:35.910031524Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O60kezSGyfXCzbn8fOxwPfi","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.913589_189.1","section":"bucket","name":"cngrzz-16","timestamp":"2020-07-13T06:16:35.913589491Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O60kezSGyfXCzbn8fOxwPfi","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.938308_190.1","section":"bucket.instance","name":"cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26","timestamp":"2020-07-13T06:16:35.938308264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.942759_191.1","section":"bucket.instance","name":"cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26","timestamp":"2020-07-13T06:16:35.942759365Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.944786_192.1","section":"bucket","name":"cngrzz-17","timestamp":"2020-07-13T06:16:35.944786921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uHYXM5KyUTxZhRfiZctWD4n","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.948764_193.1","section":"bucket","name":"cngrzz-17","timestamp":"2020-07-13T06:16:35.948764599Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uHYXM5KyUTxZhRfiZctWD4n","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.990105_194.1","section":"bucket.instance","name":"cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27","timestamp":"2020-07-13T06:16:35.990105813Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":1},"status":{"status":"write"}}},{"id":"1_1594620995.994448_195.1","section":"bucket.instance","name":"cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27","timestamp":"2020-07-13T06:16:35.994448116Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620995.996481_196.1","section":"bucket","name":"cngrzz-18","timestamp":"2020-07-13T06:16:35.996481980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z5ZtESqqP15O0oomxjuHPqh","ver":1},"status":{"status":"write"}}},{"id":"1_1594620996.000544_197.1","section":"bucket","name":"cngrzz-18","timestamp":"2020-07-13T06:16:36.000544311Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z5ZtESqqP15O0oomxjuHPqh","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620996.023617_198.1","section":"bucket.instance","name":"cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25","timestamp":"2020-07-13T06:16:36.023617389Z","data":{"read_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":1},"write_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":2},"status":{"status":"write"}}},{"id":"1_1594620996.035328_199.1","section":"bucket.instance","name":"cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25","timestamp":"2020-07-13T06:16:36.035328199Z","data":{"read_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":1},"write_version":{"tag":"__hTI4JtVEiTDJywKXIGTVi8","ver":2},"status":{"status":"complete"}}},{"id":"1_1594620996.042162_200.1","section":"bucket.instance","name":"cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26","timestamp":"2020-07-13T06:16:36.042162134Z","data":{"read_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":1},"write_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":2},"status":{"status":"write"}}},{"id":"1_1594620996.046721_201.1","section":"bucket.instance","name":"cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26","timestamp":"2020-07-13T06:16:36.046721459Z","data":{"read_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":1},"write_version":{"tag":"_eaa5TRpu1gpDEAa3Ix6y_Cf","ver":2},"status":{"status":"complete"}}},{"id":"1_1594620996.060062_202.1","section":"bucket.instance","name":"cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27","timestamp":"2020-07-13T06:16:36.060062705Z","data":{"read_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":1},"write_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":2},"status":{"status":"write"}}},{"id":"1_1594620996.066494_203.1","section":"bucket.instance","name":"cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27","timestamp":"2020-07-13T06:16:36.066494340Z","data":{"read_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":1},"write_version":{"tag":"_RmTVBDybSeWAwWkHP8zFMF_","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621170.096466_265.1","section":"bucket.instance","name":"cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28","timestamp":"2020-07-13T06:19:30.096466874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":1},"status":{"status":"write"}}},{"id":"1_1594621170.101095_286.1","section":"bucket.instance","name":"cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28","timestamp":"2020-07-13T06:19:30.101095792Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621170.103270_287.1","section":"bucket","name":"cngrzz-19","timestamp":"2020-07-13T06:19:30.103270825Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wQI4zfCNmpgzGLmtq7JYIcF","ver":1},"status":{"status":"write"}}},{"id":"1_1594621170.107053_288.1","section":"bucket","name":"cngrzz-19","timestamp":"2020-07-13T06:19:30.107053912Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wQI4zfCNmpgzGLmtq7JYIcF","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621170.117756_289.1","section":"bucket.instance","name":"cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28","timestamp":"2020-07-13T06:19:30.117756676Z","data":{"read_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":1},"write_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":2},"status":{"status":"write"}}},{"id":"1_1594621170.123767_290.1","section":"bucket.instance","name":"cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28","timestamp":"2020-07-13T06:19:30.123767219Z","data":{"read_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":1},"write_version":{"tag":"_BFKUQ6lTiElNCaecke-rbBn","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621322.052070_335.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:22:02.052070538Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":1},"status":{"status":"write"}}},{"id":"1_1594621322.057256_350.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:22:02.057256567Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621322.059913_351.1","section":"bucket","name":"cngrzz-20","timestamp":"2020-07-13T06:22:02.059913372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SiUnqH4drkKjEAXU4lM6KVi","ver":1},"status":{"status":"write"}}},{"id":"1_1594621322.065336_352.1","section":"bucket","name":"cngrzz-20","timestamp":"2020-07-13T06:22:02.065336451Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SiUnqH4drkKjEAXU4lM6KVi","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621329.476722_365.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:22:09.476722756Z","data":{"read_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":1},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":2},"status":{"status":"write"}}},{"id":"1_1594621329.487977_368.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:22:09.487977095Z","data":{"read_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":1},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621390.911893_402.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:23:10.911893325Z","data":{"read_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":2},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":3},"status":{"status":"write"}}},{"id":"1_1594621390.921870_405.1","section":"bucket.instance","name":"cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29","timestamp":"2020-07-13T06:23:10.921870742Z","data":{"read_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":2},"write_version":{"tag":"_pguz4WP27m3mCqA73M92jPp","ver":3},"status":{"status":"complete"}}},{"id":"1_1594621433.317645_430.1","section":"bucket.instance","name":"cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30","timestamp":"2020-07-13T06:23:53.317645856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":1},"status":{"status":"write"}}},{"id":"1_1594621433.322934_433.1","section":"bucket.instance","name":"cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30","timestamp":"2020-07-13T06:23:53.322934869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621433.324814_434.1","section":"bucket","name":"cngrzz-21","timestamp":"2020-07-13T06:23:53.324814398Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vqw1RgtzigBqWBzMKd9qNek","ver":1},"status":{"status":"write"}}},{"id":"1_1594621433.328408_435.1","section":"bucket","name":"cngrzz-21","timestamp":"2020-07-13T06:23:53.328408882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vqw1RgtzigBqWBzMKd9qNek","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621433.386628_436.1","section":"bucket.instance","name":"cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31","timestamp":"2020-07-13T06:23:53.386628999Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":1},"status":{"status":"write"}}},{"id":"1_1594621433.390873_437.1","section":"bucket.instance","name":"cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31","timestamp":"2020-07-13T06:23:53.390873686Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621433.393237_438.1","section":"bucket","name":"cngrzz-22","timestamp":"2020-07-13T06:23:53.393237663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7bwonnuvmzNIIPT8mR6P0bA","ver":1},"status":{"status":"write"}}},{"id":"1_1594621433.397692_439.1","section":"bucket","name":"cngrzz-22","timestamp":"2020-07-13T06:23:53.397692663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7bwonnuvmzNIIPT8mR6P0bA","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621434.232311_440.1","section":"bucket.instance","name":"cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32","timestamp":"2020-07-13T06:23:54.232311850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":1},"status":{"status":"write"}}},{"id":"1_1594621434.236794_441.1","section":"bucket.instance","name":"cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32","timestamp":"2020-07-13T06:23:54.236794211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621434.238800_442.1","section":"bucket","name":"cngrzz-23","timestamp":"2020-07-13T06:23:54.238800428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SvlhobBY9Slsbwne1CTEC-L","ver":1},"status":{"status":"write"}}},{"id":"1_1594621434.242788_443.1","section":"bucket","name":"cngrzz-23","timestamp":"2020-07-13T06:23:54.242788736Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SvlhobBY9Slsbwne1CTEC-L","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621434.264342_444.1","section":"bucket.instance","name":"cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30","timestamp":"2020-07-13T06:23:54.264342494Z","data":{"read_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":1},"write_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":2},"status":{"status":"write"}}},{"id":"1_1594621434.270547_445.1","section":"bucket.instance","name":"cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30","timestamp":"2020-07-13T06:23:54.270547881Z","data":{"read_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":1},"write_version":{"tag":"_-pFeDPf9Eat5X27AAB2zFlB","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621434.275368_446.1","section":"bucket.instance","name":"cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31","timestamp":"2020-07-13T06:23:54.275368951Z","data":{"read_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":1},"write_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":2},"status":{"status":"write"}}},{"id":"1_1594621434.282613_447.1","section":"bucket.instance","name":"cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31","timestamp":"2020-07-13T06:23:54.282613618Z","data":{"read_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":1},"write_version":{"tag":"_aCaPPvnYfC8irtNCrLOm49i","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621434.294881_448.1","section":"bucket.instance","name":"cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32","timestamp":"2020-07-13T06:23:54.294881064Z","data":{"read_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":1},"write_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":2},"status":{"status":"write"}}},{"id":"1_1594621434.303397_449.1","section":"bucket.instance","name":"cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32","timestamp":"2020-07-13T06:23:54.303397214Z","data":{"read_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":1},"write_version":{"tag":"_ODHU4Sro1-VxPNEqDU49Dhb","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621455.307113_481.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.307113064Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.311789_492.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.311789146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.314390_493.1","section":"bucket","name":"cngrzz-24","timestamp":"2020-07-13T06:24:15.314390695Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_syYOTmiVtbmFjOCG8p3NuzV","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.318379_494.1","section":"bucket","name":"cngrzz-24","timestamp":"2020-07-13T06:24:15.318379915Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_syYOTmiVtbmFjOCG8p3NuzV","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.344614_495.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.344614344Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.349148_496.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.349148544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.351560_497.1","section":"bucket","name":"cngrzz-25","timestamp":"2020-07-13T06:24:15.351560923Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L6eQ90-G-twMkpkPL-m5q_J","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.355886_498.1","section":"bucket","name":"cngrzz-25","timestamp":"2020-07-13T06:24:15.355886460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L6eQ90-G-twMkpkPL-m5q_J","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.405945_499.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.405945564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.410863_500.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.410863468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.412859_501.1","section":"bucket","name":"cngrzz-26","timestamp":"2020-07-13T06:24:15.412859756Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wVuwzieS4x5WATempXhxurc","ver":1},"status":{"status":"write"}}},{"id":"1_1594621455.417215_502.1","section":"bucket","name":"cngrzz-26","timestamp":"2020-07-13T06:24:15.417215461Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wVuwzieS4x5WATempXhxurc","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621455.441624_503.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.441624712Z","data":{"read_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":1},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":2},"status":{"status":"write"}}},{"id":"1_1594621455.464024_504.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.464024694Z","data":{"read_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":1},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621455.470869_505.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.470869735Z","data":{"read_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":2},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":3},"status":{"status":"write"}}},{"id":"1_1594621455.475511_506.1","section":"bucket.instance","name":"cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33","timestamp":"2020-07-13T06:24:15.475511044Z","data":{"read_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":2},"write_version":{"tag":"_k7k2o-rnDQA99BzADhSUOc_","ver":3},"status":{"status":"complete"}}},{"id":"1_1594621455.479588_507.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.479588113Z","data":{"read_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":1},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":2},"status":{"status":"write"}}},{"id":"1_1594621455.484136_508.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.484136677Z","data":{"read_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":1},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621455.494806_509.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.494806032Z","data":{"read_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":2},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":3},"status":{"status":"write"}}},{"id":"1_1594621455.505325_510.1","section":"bucket.instance","name":"cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34","timestamp":"2020-07-13T06:24:15.505325897Z","data":{"read_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":2},"write_version":{"tag":"_6OmPnx6DYgwPh35e16VwKxB","ver":3},"status":{"status":"complete"}}},{"id":"1_1594621455.518131_511.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.518131893Z","data":{"read_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":1},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":2},"status":{"status":"write"}}},{"id":"1_1594621455.527513_512.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.527513994Z","data":{"read_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":1},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621455.542157_513.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.542157223Z","data":{"read_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":2},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":3},"status":{"status":"write"}}},{"id":"1_1594621455.551984_514.1","section":"bucket.instance","name":"cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35","timestamp":"2020-07-13T06:24:15.551984028Z","data":{"read_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":2},"write_version":{"tag":"_7FBTpEO4-5rUUoWqeRM3U8I","ver":3},"status":{"status":"complete"}}},{"id":"1_1594621478.463758_549.1","section":"bucket.instance","name":"cngrzz-27:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.36","timestamp":"2020-07-13T06:24:38.463758191Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.468096_556.1","section":"bucket.instance","name":"cngrzz-27:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.36","timestamp":"2020-07-13T06:24:38.468096388Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.470142_557.1","section":"bucket","name":"cngrzz-27","timestamp":"2020-07-13T06:24:38.470142675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xKmuAPoskvLbXPU03OlC73s","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.473856_558.1","section":"bucket","name":"cngrzz-27","timestamp":"2020-07-13T06:24:38.473856655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xKmuAPoskvLbXPU03OlC73s","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.497896_559.1","section":"bucket.instance","name":"cngrzz-28:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.37","timestamp":"2020-07-13T06:24:38.497896309Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.502362_560.1","section":"bucket.instance","name":"cngrzz-28:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.37","timestamp":"2020-07-13T06:24:38.502362092Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.504706_561.1","section":"bucket","name":"cngrzz-28","timestamp":"2020-07-13T06:24:38.504706153Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m0vuxwdxcV4K0zwl3eB9Wa9","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.509299_562.1","section":"bucket","name":"cngrzz-28","timestamp":"2020-07-13T06:24:38.509299001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m0vuxwdxcV4K0zwl3eB9Wa9","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.554287_563.1","section":"bucket.instance","name":"cngrzz-29:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.38","timestamp":"2020-07-13T06:24:38.554287987Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.561718_564.1","section":"bucket.instance","name":"cngrzz-29:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.38","timestamp":"2020-07-13T06:24:38.561718878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.564680_565.1","section":"bucket","name":"cngrzz-29","timestamp":"2020-07-13T06:24:38.564680554Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2TuYKigZrzuv-dMOyxValfc","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.569842_566.1","section":"bucket","name":"cngrzz-29","timestamp":"2020-07-13T06:24:38.569842326Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2TuYKigZrzuv-dMOyxValfc","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.595037_567.1","section":"bucket.instance","name":"cngrzz-27:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.36","timestamp":"2020-07-13T06:24:38.595037740Z","data":{"read_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":1},"write_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":2},"status":{"status":"write"}}},{"id":"1_1594621478.600762_568.1","section":"bucket.instance","name":"cngrzz-27:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.36","timestamp":"2020-07-13T06:24:38.600762996Z","data":{"read_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":1},"write_version":{"tag":"_tMoerZ70TfsAfUnMO2ZU0BQ","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621478.647960_569.1","section":"bucket.instance","name":"cngrzz-28:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.37","timestamp":"2020-07-13T06:24:38.647960510Z","data":{"read_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":1},"write_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":2},"status":{"status":"write"}}},{"id":"1_1594621478.658940_570.1","section":"bucket.instance","name":"cngrzz-28:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.37","timestamp":"2020-07-13T06:24:38.658940233Z","data":{"read_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":1},"write_version":{"tag":"_5JLsGEdYmhtceOraexKNZHW","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621478.723174_571.1","section":"bucket.instance","name":"cngrzz-29:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.38","timestamp":"2020-07-13T06:24:38.723174021Z","data":{"read_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":1},"write_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":2},"status":{"status":"write"}}},{"id":"1_1594621478.729364_572.1","section":"bucket.instance","name":"cngrzz-29:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.38","timestamp":"2020-07-13T06:24:38.729364605Z","data":{"read_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":1},"write_version":{"tag":"_ON3wSOtBnL78r4YJaBVEeNR","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621478.803680_573.1","section":"bucket.instance","name":"cngrzz-30:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.39","timestamp":"2020-07-13T06:24:38.803680169Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.808550_574.1","section":"bucket.instance","name":"cngrzz-30:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.39","timestamp":"2020-07-13T06:24:38.808550868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.810788_575.1","section":"bucket","name":"cngrzz-30","timestamp":"2020-07-13T06:24:38.810788109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tcbDPAiLUbg1Qosf4aVJZaR","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.814964_576.1","section":"bucket","name":"cngrzz-30","timestamp":"2020-07-13T06:24:38.814964209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tcbDPAiLUbg1Qosf4aVJZaR","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.842441_577.1","section":"bucket.instance","name":"cngrzz-31:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.40","timestamp":"2020-07-13T06:24:38.842441698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.846740_578.1","section":"bucket.instance","name":"cngrzz-31:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.40","timestamp":"2020-07-13T06:24:38.846740063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.849242_579.1","section":"bucket","name":"cngrzz-31","timestamp":"2020-07-13T06:24:38.849242603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PcNMXYzKJiqzUL9v_g9UUEW","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.853963_580.1","section":"bucket","name":"cngrzz-31","timestamp":"2020-07-13T06:24:38.853963184Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PcNMXYzKJiqzUL9v_g9UUEW","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.938346_581.1","section":"bucket.instance","name":"cngrzz-32:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.41","timestamp":"2020-07-13T06:24:38.938346254Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.942593_582.1","section":"bucket.instance","name":"cngrzz-32:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.41","timestamp":"2020-07-13T06:24:38.942593554Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.944644_583.1","section":"bucket","name":"cngrzz-32","timestamp":"2020-07-13T06:24:38.944644272Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cgRpl2sxiftnVUa2OO_gEgo","ver":1},"status":{"status":"write"}}},{"id":"1_1594621478.948537_584.1","section":"bucket","name":"cngrzz-32","timestamp":"2020-07-13T06:24:38.948537812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cgRpl2sxiftnVUa2OO_gEgo","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621478.974066_585.1","section":"bucket.instance","name":"cngrzz-30:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.39","timestamp":"2020-07-13T06:24:38.974066414Z","data":{"read_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":1},"write_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":2},"status":{"status":"write"}}},{"id":"1_1594621478.980780_586.1","section":"bucket.instance","name":"cngrzz-30:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.39","timestamp":"2020-07-13T06:24:38.980780610Z","data":{"read_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":1},"write_version":{"tag":"_6RbOb0wgKTZytbclwaadvIb","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621479.033952_587.1","section":"bucket.instance","name":"cngrzz-31:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.40","timestamp":"2020-07-13T06:24:39.033952236Z","data":{"read_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":1},"write_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":2},"status":{"status":"write"}}},{"id":"1_1594621479.045331_588.1","section":"bucket.instance","name":"cngrzz-31:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.40","timestamp":"2020-07-13T06:24:39.045331654Z","data":{"read_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":1},"write_version":{"tag":"_SL4wmneHEMqLXcyG1BUFxkV","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621479.109134_589.1","section":"bucket.instance","name":"cngrzz-32:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.41","timestamp":"2020-07-13T06:24:39.109134981Z","data":{"read_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":1},"write_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":2},"status":{"status":"write"}}},{"id":"1_1594621479.118646_590.1","section":"bucket.instance","name":"cngrzz-32:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.41","timestamp":"2020-07-13T06:24:39.118646708Z","data":{"read_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":1},"write_version":{"tag":"_6V3_gtxchXIPzjreXBaOuYI","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621479.194094_591.1","section":"bucket.instance","name":"cngrzz-33:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.42","timestamp":"2020-07-13T06:24:39.194094933Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.200722_592.1","section":"bucket.instance","name":"cngrzz-33:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.42","timestamp":"2020-07-13T06:24:39.200722310Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.203027_593.1","section":"bucket","name":"cngrzz-33","timestamp":"2020-07-13T06:24:39.203027753Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PuawPXsI3tXqBy_MXXvr0Cy","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.209920_594.1","section":"bucket","name":"cngrzz-33","timestamp":"2020-07-13T06:24:39.209920207Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PuawPXsI3tXqBy_MXXvr0Cy","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.241265_595.1","section":"bucket.instance","name":"cngrzz-34:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.43","timestamp":"2020-07-13T06:24:39.241265494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.246173_596.1","section":"bucket.instance","name":"cngrzz-34:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.43","timestamp":"2020-07-13T06:24:39.246173690Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.248646_597.1","section":"bucket","name":"cngrzz-34","timestamp":"2020-07-13T06:24:39.248646104Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SZllY3UvM6YWJqOBtYiiacb","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.253617_598.1","section":"bucket","name":"cngrzz-34","timestamp":"2020-07-13T06:24:39.253617145Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SZllY3UvM6YWJqOBtYiiacb","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.323567_599.1","section":"bucket.instance","name":"cngrzz-35:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.44","timestamp":"2020-07-13T06:24:39.323567314Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.328276_600.1","section":"bucket.instance","name":"cngrzz-35:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.44","timestamp":"2020-07-13T06:24:39.328276870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.330806_601.1","section":"bucket","name":"cngrzz-35","timestamp":"2020-07-13T06:24:39.330806173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v-R90_jF6GVW73YIo8cHfq0","ver":1},"status":{"status":"write"}}},{"id":"1_1594621479.336092_602.1","section":"bucket","name":"cngrzz-35","timestamp":"2020-07-13T06:24:39.336092549Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v-R90_jF6GVW73YIo8cHfq0","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621479.359801_603.1","section":"bucket.instance","name":"cngrzz-33:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.42","timestamp":"2020-07-13T06:24:39.359801153Z","data":{"read_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":1},"write_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":2},"status":{"status":"write"}}},{"id":"1_1594621479.369370_604.1","section":"bucket.instance","name":"cngrzz-33:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.42","timestamp":"2020-07-13T06:24:39.369370295Z","data":{"read_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":1},"write_version":{"tag":"_dpMT8NJhyR0KhLLSeNDva5A","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621480.420657_605.1","section":"bucket.instance","name":"cngrzz-34:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.43","timestamp":"2020-07-13T06:24:40.420657898Z","data":{"read_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":1},"write_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":2},"status":{"status":"write"}}},{"id":"1_1594621480.432601_606.1","section":"bucket.instance","name":"cngrzz-34:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.43","timestamp":"2020-07-13T06:24:40.432601102Z","data":{"read_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":1},"write_version":{"tag":"_B9dqcoWfU3H8SwLzPF22u5w","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621482.474186_607.1","section":"bucket.instance","name":"cngrzz-35:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.44","timestamp":"2020-07-13T06:24:42.474186915Z","data":{"read_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":1},"write_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":2},"status":{"status":"write"}}},{"id":"1_1594621482.484822_608.1","section":"bucket.instance","name":"cngrzz-35:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.44","timestamp":"2020-07-13T06:24:42.484822401Z","data":{"read_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":1},"write_version":{"tag":"_-56ZepAIsfzcBN18Tslcasc","ver":2},"status":{"status":"complete"}}},{"id":"1_1594621482.913401_609.1","section":"bucket.instance","name":"cngrzz-36:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.45","timestamp":"2020-07-13T06:24:42.913401257Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BHYIPGqLECVNEi2cZ-MYjOz","ver":1},"status":{"status":"write"}}},{"id":"1_1594621482.917800_610.1","section":"bucket.instance","name":"cngrzz-36:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.45","timestamp":"2020-07-13T06:24:42.917800056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BHYIPGqLECVNEi2cZ-MYjOz","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621482.920043_611.1","section":"bucket","name":"cngrzz-36","timestamp":"2020-07-13T06:24:42.920043693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lDf1FKkxnRubfsQ9M1--dgD","ver":1},"status":{"status":"write"}}},{"id":"1_1594621482.924506_612.1","section":"bucket","name":"cngrzz-36","timestamp":"2020-07-13T06:24:42.924506470Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lDf1FKkxnRubfsQ9M1--dgD","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621483.015114_613.1","section":"bucket.instance","name":"cngrzz-37:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.46","timestamp":"2020-07-13T06:24:43.015114675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oP_SUc8PxL6c4XMdnGrcIDd","ver":1},"status":{"status":"write"}}},{"id":"1_1594621483.020538_614.1","section":"bucket.instance","name":"cngrzz-37:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.46","timestamp":"2020-07-13T06:24:43.020538172Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oP_SUc8PxL6c4XMdnGrcIDd","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621483.022545_615.1","section":"bucket","name":"cngrzz-37","timestamp":"2020-07-13T06:24:43.022545950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5gFmRJU_Lnfca_c3ktRTso7","ver":1},"status":{"status":"write"}}},{"id":"1_1594621483.026371_616.1","section":"bucket","name":"cngrzz-37","timestamp":"2020-07-13T06:24:43.026371722Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5gFmRJU_Lnfca_c3ktRTso7","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621483.081477_617.1","section":"bucket.instance","name":"cngrzz-38:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.47","timestamp":"2020-07-13T06:24:43.081477070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nVm3w2ql0Kn2oqc2UsEJH8a","ver":1},"status":{"status":"write"}}},{"id":"1_1594621483.087242_618.1","section":"bucket.instance","name":"cngrzz-38:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.47","timestamp":"2020-07-13T06:24:43.087242473Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nVm3w2ql0Kn2oqc2UsEJH8a","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621483.089407_619.1","section":"bucket","name":"cngrzz-38","timestamp":"2020-07-13T06:24:43.089407632Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3eyVOL0MMcQBz9oFqM87zxz","ver":1},"status":{"status":"write"}}},{"id":"1_1594621483.093117_620.1","section":"bucket","name":"cngrzz-38","timestamp":"2020-07-13T06:24:43.093117980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3eyVOL0MMcQBz9oFqM87zxz","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.461368_621.1","section":"bucket.instance","name":"cngrzz-39:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.48","timestamp":"2020-07-13T06:24:46.461368112Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uMnm_CbH26xOW2j2VokpY4s","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.466851_624.1","section":"bucket.instance","name":"cngrzz-39:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.48","timestamp":"2020-07-13T06:24:46.466851306Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uMnm_CbH26xOW2j2VokpY4s","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.469013_625.1","section":"bucket","name":"cngrzz-39","timestamp":"2020-07-13T06:24:46.469013868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eqEaW0o9d4QYozEF28u7c49","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.472599_626.1","section":"bucket","name":"cngrzz-39","timestamp":"2020-07-13T06:24:46.472599369Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eqEaW0o9d4QYozEF28u7c49","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.759727_627.1","section":"bucket.instance","name":"cngrzz-40:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.49","timestamp":"2020-07-13T06:24:46.759727960Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OP3txqbvev88w0dP9vK-dO5","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.765631_628.1","section":"bucket.instance","name":"cngrzz-40:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.49","timestamp":"2020-07-13T06:24:46.765631411Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OP3txqbvev88w0dP9vK-dO5","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.767996_629.1","section":"bucket","name":"cngrzz-40","timestamp":"2020-07-13T06:24:46.767996150Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M3LKRjOIblMfECEncxsHzyu","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.772433_630.1","section":"bucket","name":"cngrzz-40","timestamp":"2020-07-13T06:24:46.772433975Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M3LKRjOIblMfECEncxsHzyu","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.817605_631.1","section":"bucket.instance","name":"cngrzz-41:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.50","timestamp":"2020-07-13T06:24:46.817605518Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kHTWaobdOcV3WvsIXS-4JDw","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.823059_632.1","section":"bucket.instance","name":"cngrzz-41:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.50","timestamp":"2020-07-13T06:24:46.823059818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kHTWaobdOcV3WvsIXS-4JDw","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621486.825455_633.1","section":"bucket","name":"cngrzz-41","timestamp":"2020-07-13T06:24:46.825455311Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_enzpzPLLpAzWWe8NZI6ZjKh","ver":1},"status":{"status":"write"}}},{"id":"1_1594621486.829707_634.1","section":"bucket","name":"cngrzz-41","timestamp":"2020-07-13T06:24:46.829707023Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_enzpzPLLpAzWWe8NZI6ZjKh","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620914.565843_67.1","section":"bucket.instance","name":"cngrzz-1:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.1","timestamp":"2020-07-13T06:15:14.565843290Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NrX7NUBlKlctPxLgZQhVZeJ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620914.569624_68.1","section":"bucket.instance","name":"cngrzz-1:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.1","timestamp":"2020-07-13T06:15:14.569624562Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NrX7NUBlKlctPxLgZQhVZeJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620914.571621_69.1","section":"bucket","name":"cngrzz-1","timestamp":"2020-07-13T06:15:14.571621151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B_xEGfjqp5GJf-kckzrymxy","ver":1},"status":{"status":"write"}}},{"id":"1_1594620914.576799_70.1","section":"bucket","name":"cngrzz-1","timestamp":"2020-07-13T06:15:14.576799306Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B_xEGfjqp5GJf-kckzrymxy","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620914.604902_71.1","section":"bucket.instance","name":"cngrzz-2:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.2","timestamp":"2020-07-13T06:15:14.604902219Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PvZsoXHBEggBhfEef7o5U0T","ver":1},"status":{"status":"write"}}},{"id":"1_1594620914.609491_72.1","section":"bucket.instance","name":"cngrzz-2:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.2","timestamp":"2020-07-13T06:15:14.609491570Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PvZsoXHBEggBhfEef7o5U0T","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620914.612054_73.1","section":"bucket","name":"cngrzz-2","timestamp":"2020-07-13T06:15:14.612054698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6IT0MMiYAzd8ADA6mJQGl2V","ver":1},"status":{"status":"write"}}},{"id":"1_1594620914.620627_74.1","section":"bucket","name":"cngrzz-2","timestamp":"2020-07-13T06:15:14.620627832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6IT0MMiYAzd8ADA6mJQGl2V","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620918.616085_75.1","section":"bucket.instance","name":"cngrzz-3:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.3","timestamp":"2020-07-13T06:15:18.616085899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jNBNF8gIfCQ0VtU1Nd6VsXo","ver":1},"status":{"status":"write"}}},{"id":"1_1594620918.620674_76.1","section":"bucket.instance","name":"cngrzz-3:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.3","timestamp":"2020-07-13T06:15:18.620674736Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jNBNF8gIfCQ0VtU1Nd6VsXo","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620918.625187_77.1","section":"bucket","name":"cngrzz-3","timestamp":"2020-07-13T06:15:18.625187717Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6UxFHip67VoBBxh6FLU5ArZ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620918.629897_78.1","section":"bucket","name":"cngrzz-3","timestamp":"2020-07-13T06:15:18.629897499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6UxFHip67VoBBxh6FLU5ArZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.868689_79.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.4","timestamp":"2020-07-13T06:15:21.868689028Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5eRMS50W9omalXkI-eKEkZH","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.873048_80.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.4","timestamp":"2020-07-13T06:15:21.873048233Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5eRMS50W9omalXkI-eKEkZH","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.874813_81.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:21.874813555Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JboyausbemE9wke0D-OFRJL","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.881925_82.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:21.881925098Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JboyausbemE9wke0D-OFRJL","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.904963_83.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.5","timestamp":"2020-07-13T06:15:21.904963978Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_orhbCdPThQDntboRt2h0Sl9","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.910649_84.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.5","timestamp":"2020-07-13T06:15:21.910649089Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_orhbCdPThQDntboRt2h0Sl9","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.913196_85.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:21.913196616Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W9AKCYVnn-UeTSsMsys7MT_","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.920774_86.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:21.920774888Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W9AKCYVnn-UeTSsMsys7MT_","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.970374_87.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.6","timestamp":"2020-07-13T06:15:21.970374493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v6ZB9vPFQIwn4oxtloegK6s","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.976019_88.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.6","timestamp":"2020-07-13T06:15:21.976019891Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v6ZB9vPFQIwn4oxtloegK6s","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620921.980716_89.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:21.980716496Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IjM-H8ZOw8IgEVRDW0EGVzm","ver":1},"status":{"status":"write"}}},{"id":"1_1594620921.991312_90.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:21.991312189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IjM-H8ZOw8IgEVRDW0EGVzm","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.389888_91.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.7","timestamp":"2020-07-13T06:15:25.389888371Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RWdhBH265JDcuhXxt8yJvwV","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.394176_92.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.7","timestamp":"2020-07-13T06:15:25.394176539Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RWdhBH265JDcuhXxt8yJvwV","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.396386_93.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:25.396386821Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t0iFgy3ChisA7Hu7SEB54i8","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.400663_94.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:25.400663561Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_t0iFgy3ChisA7Hu7SEB54i8","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.431105_95.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.8","timestamp":"2020-07-13T06:15:25.431105188Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OXxKeva7ZQcmEiQXaZBhcee","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.435395_96.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.8","timestamp":"2020-07-13T06:15:25.435395684Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OXxKeva7ZQcmEiQXaZBhcee","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.440261_97.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:25.440261819Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IbupanJOb1vHvGDroO9OyV1","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.444849_98.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:25.444849981Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IbupanJOb1vHvGDroO9OyV1","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.475021_99.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.9","timestamp":"2020-07-13T06:15:25.475021111Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sws_fFW3wnseWgGw3cH1pX-","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.479177_100.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.9","timestamp":"2020-07-13T06:15:25.479177003Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sws_fFW3wnseWgGw3cH1pX-","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.481306_101.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:25.481306318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Lq9uI5UBhcdqocC2jNTLpQ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.486249_102.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:25.486249916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Lq9uI5UBhcdqocC2jNTLpQ","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.679609_103.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.10","timestamp":"2020-07-13T06:15:25.679609508Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mlzW7asaUOM35pXKhvz5VX2","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.684249_104.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.10","timestamp":"2020-07-13T06:15:25.684249287Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mlzW7asaUOM35pXKhvz5VX2","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.686238_105.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:25.686238812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IRKdAJCiQhoxy_hbM2Zp4rZ","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.690770_106.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:25.690770954Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IRKdAJCiQhoxy_hbM2Zp4rZ","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.750744_107.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.11","timestamp":"2020-07-13T06:15:25.750744600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TVxzlNwS0AToOujH53HcOeO","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.755250_108.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.11","timestamp":"2020-07-13T06:15:25.755250926Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TVxzlNwS0AToOujH53HcOeO","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.759191_109.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:25.759191648Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__V6H1e1TkzH83WDQjMnLs3J","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.763788_110.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:25.763788507Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__V6H1e1TkzH83WDQjMnLs3J","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620925.874919_111.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.12","timestamp":"2020-07-13T06:15:25.874919330Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cppER_j32QTT6jfkUc7OtN9","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.880056_112.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.12","timestamp":"2020-07-13T06:15:25.880056550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cppER_j32QTT6jfkUc7OtN9","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620925.882580_113.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:25.882580782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1kbjCgsQlkaQET_-cKcofT8","ver":1},"status":{"status":"write"}}},{"id":"1_1594620925.887578_114.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:25.887578813Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1kbjCgsQlkaQET_-cKcofT8","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620926.372198_115.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.13","timestamp":"2020-07-13T06:15:26.372198300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AbGEn19L1mso3vjlMICZuzH","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.376648_116.1","section":"bucket.instance","name":"cngrzz-4:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.13","timestamp":"2020-07-13T06:15:26.376648564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AbGEn19L1mso3vjlMICZuzH","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620926.379137_117.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:26.379137187Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AJ1-E53HupEVn1PaRJLxknN","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.385397_118.1","section":"bucket","name":"cngrzz-4","timestamp":"2020-07-13T06:15:26.385397048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AJ1-E53HupEVn1PaRJLxknN","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620926.440877_119.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.14","timestamp":"2020-07-13T06:15:26.440877932Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QwRkzlppS0ZPFANJbrxRLAo","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.445773_120.1","section":"bucket.instance","name":"cngrzz-5:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.14","timestamp":"2020-07-13T06:15:26.445773115Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QwRkzlppS0ZPFANJbrxRLAo","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620926.447974_121.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:26.447974475Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sXT0pkowqKazgGp3FXuQLsA","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.452307_122.1","section":"bucket","name":"cngrzz-5","timestamp":"2020-07-13T06:15:26.452307296Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sXT0pkowqKazgGp3FXuQLsA","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620926.558326_123.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.15","timestamp":"2020-07-13T06:15:26.558326355Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OrXmOtQEOSJJK03Tf2okhoA","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.563257_124.1","section":"bucket.instance","name":"cngrzz-6:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.15","timestamp":"2020-07-13T06:15:26.563257650Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OrXmOtQEOSJJK03Tf2okhoA","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620926.565740_125.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:26.565740914Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AF1yTvVufSmQzFmQD6m6PPG","ver":1},"status":{"status":"write"}}},{"id":"1_1594620926.570739_126.1","section":"bucket","name":"cngrzz-6","timestamp":"2020-07-13T06:15:26.570739160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AF1yTvVufSmQzFmQD6m6PPG","ver":1},"status":{"status":"abort"}}},{"id":"1_1594620929.001475_127.1","section":"bucket.instance","name":"cngrzz-7:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.16","timestamp":"2020-07-13T06:15:29.001475256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ng8rN4G68wRQ3eDQZ46UDUy","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.006047_128.1","section":"bucket.instance","name":"cngrzz-7:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.16","timestamp":"2020-07-13T06:15:29.006047645Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ng8rN4G68wRQ3eDQZ46UDUy","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.008050_129.1","section":"bucket","name":"cngrzz-7","timestamp":"2020-07-13T06:15:29.008050687Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.012811_130.1","section":"bucket","name":"cngrzz-7","timestamp":"2020-07-13T06:15:29.012811406Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.036306_131.1","section":"bucket.instance","name":"cngrzz-8:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.17","timestamp":"2020-07-13T06:15:29.036306449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AreU9GaZl3bJuU19nqk4OSY","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.040693_132.1","section":"bucket.instance","name":"cngrzz-8:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.17","timestamp":"2020-07-13T06:15:29.040693341Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AreU9GaZl3bJuU19nqk4OSY","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.042995_133.1","section":"bucket","name":"cngrzz-8","timestamp":"2020-07-13T06:15:29.042995034Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.047392_134.1","section":"bucket","name":"cngrzz-8","timestamp":"2020-07-13T06:15:29.047392137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.090400_135.1","section":"bucket.instance","name":"cngrzz-9:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.18","timestamp":"2020-07-13T06:15:29.090400277Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k8n1zW2nPOQWnZG2f7x7kJc","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.094794_136.1","section":"bucket.instance","name":"cngrzz-9:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.18","timestamp":"2020-07-13T06:15:29.094794643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k8n1zW2nPOQWnZG2f7x7kJc","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620929.096824_137.1","section":"bucket","name":"cngrzz-9","timestamp":"2020-07-13T06:15:29.096824917Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":1},"status":{"status":"write"}}},{"id":"1_1594620929.100492_138.1","section":"bucket","name":"cngrzz-9","timestamp":"2020-07-13T06:15:29.100492014Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":1},"status":{"status":"complete"}}},{"id":"1_1594620932.680034_139.1","section":"bucket","name":"cngrzz-7","timestamp":"2020-07-13T06:15:32.680034714Z","data":{"read_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":1},"write_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":2},"status":{"status":"remove"}}},{"id":"1_1594620932.686075_140.1","section":"bucket","name":"cngrzz-7","timestamp":"2020-07-13T06:15:32.686075353Z","data":{"read_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":1},"write_version":{"tag":"_ajE2RendXZSmdBOu-R7O7xz","ver":2},"status":{"status":"complete"}}},{"id":"1_1594620933.010327_141.1","section":"bucket","name":"cngrzz-8","timestamp":"2020-07-13T06:15:33.010327021Z","data":{"read_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":1},"write_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":2},"status":{"status":"remove"}}},{"id":"1_1594620933.017231_142.1","section":"bucket","name":"cngrzz-8","timestamp":"2020-07-13T06:15:33.017231115Z","data":{"read_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":1},"write_version":{"tag":"_4qh0Ob95eS7aOk-HwbVNE0B","ver":2},"status":{"status":"complete"}}},{"id":"1_1594620933.436438_143.1","section":"bucket","name":"cngrzz-9","timestamp":"2020-07-13T06:15:33.436438596Z","data":{"read_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":1},"write_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":2},"status":{"status":"remove"}}},{"id":"1_1594620933.443793_144.1","section":"bucket","name":"cngrzz-9","timestamp":"2020-07-13T06:15:33.443793026Z","data":{"read_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":1},"write_version":{"tag":"_IdQw1HjgJDelEI1clGd-_Kf","ver":2},"status":{"status":"complete"}}}] 2020-07-13T06:27:06.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'c0ad3871-076a-4736-ba2c-6cf14ab7b93e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:06.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'c0ad3871-076a-4736-ba2c-6cf14ab7b93e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:06.359 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period c0ad3871-076a-4736-ba2c-6cf14ab7b93e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:06.398 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:06.398 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:06.641 INFO:teuthology.orchestra.run.smithi017.stdout:[{"id":"1_1594621554.610004_416.1","section":"bucket.instance","name":"cngrzz-42:7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.1","timestamp":"2020-07-13T06:25:54.610004858Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fg3EM4vVmVLMzqreKfIqADg","ver":1},"status":{"status":"write"}}},{"id":"1_1594621554.615406_417.1","section":"bucket.instance","name":"cngrzz-42:7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.1","timestamp":"2020-07-13T06:25:54.615406500Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fg3EM4vVmVLMzqreKfIqADg","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621554.617393_418.1","section":"bucket","name":"cngrzz-42","timestamp":"2020-07-13T06:25:54.617393527Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b8eCpo2wyeyoUHzcMY8mjUW","ver":1},"status":{"status":"write"}}},{"id":"1_1594621554.621183_419.1","section":"bucket","name":"cngrzz-42","timestamp":"2020-07-13T06:25:54.621183763Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b8eCpo2wyeyoUHzcMY8mjUW","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621554.669155_420.1","section":"bucket.instance","name":"cngrzz-43:7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.2","timestamp":"2020-07-13T06:25:54.669155022Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pruOl0R70hBg18xsfAzI4L_","ver":1},"status":{"status":"write"}}},{"id":"1_1594621554.673262_421.1","section":"bucket.instance","name":"cngrzz-43:7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.2","timestamp":"2020-07-13T06:25:54.673262872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pruOl0R70hBg18xsfAzI4L_","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621554.675281_422.1","section":"bucket","name":"cngrzz-43","timestamp":"2020-07-13T06:25:54.675281393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BQ_fGwBgvGz-aNmWMtyO2ml","ver":1},"status":{"status":"write"}}},{"id":"1_1594621554.679151_423.1","section":"bucket","name":"cngrzz-43","timestamp":"2020-07-13T06:25:54.679151359Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BQ_fGwBgvGz-aNmWMtyO2ml","ver":1},"status":{"status":"complete"}}}] 2020-07-13T06:27:06.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'c0ad3871-076a-4736-ba2c-6cf14ab7b93e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:06.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'c0ad3871-076a-4736-ba2c-6cf14ab7b93e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:06.644 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period c0ad3871-076a-4736-ba2c-6cf14ab7b93e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:06.688 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:06.688 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:06.932 INFO:teuthology.orchestra.run.smithi014.stdout:[{"id":"1_1594621554.610004_416.1","section":"bucket.instance","name":"cngrzz-42:7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.1","timestamp":"2020-07-13T06:25:54.610004858Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fg3EM4vVmVLMzqreKfIqADg","ver":1},"status":{"status":"write"}}},{"id":"1_1594621554.615406_417.1","section":"bucket.instance","name":"cngrzz-42:7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.1","timestamp":"2020-07-13T06:25:54.615406500Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fg3EM4vVmVLMzqreKfIqADg","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621554.617393_418.1","section":"bucket","name":"cngrzz-42","timestamp":"2020-07-13T06:25:54.617393527Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b8eCpo2wyeyoUHzcMY8mjUW","ver":1},"status":{"status":"write"}}},{"id":"1_1594621554.621183_419.1","section":"bucket","name":"cngrzz-42","timestamp":"2020-07-13T06:25:54.621183763Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b8eCpo2wyeyoUHzcMY8mjUW","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621554.669155_420.1","section":"bucket.instance","name":"cngrzz-43:7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.2","timestamp":"2020-07-13T06:25:54.669155022Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pruOl0R70hBg18xsfAzI4L_","ver":1},"status":{"status":"write"}}},{"id":"1_1594621554.673262_421.1","section":"bucket.instance","name":"cngrzz-43:7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.2","timestamp":"2020-07-13T06:25:54.673262872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pruOl0R70hBg18xsfAzI4L_","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621554.675281_422.1","section":"bucket","name":"cngrzz-43","timestamp":"2020-07-13T06:25:54.675281393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BQ_fGwBgvGz-aNmWMtyO2ml","ver":1},"status":{"status":"write"}}},{"id":"1_1594621554.679151_423.1","section":"bucket","name":"cngrzz-43","timestamp":"2020-07-13T06:25:54.679151359Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BQ_fGwBgvGz-aNmWMtyO2ml","ver":1},"status":{"status":"complete"}}}] 2020-07-13T06:27:06.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'c0ad3871-076a-4736-ba2c-6cf14ab7b93e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:06.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'c0ad3871-076a-4736-ba2c-6cf14ab7b93e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:06.934 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period c0ad3871-076a-4736-ba2c-6cf14ab7b93e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:06.975 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:06.975 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:07.253 INFO:teuthology.orchestra.run.smithi017.stdout:[{"id":"1_1594621554.610004_416.1","section":"bucket.instance","name":"cngrzz-42:7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.1","timestamp":"2020-07-13T06:25:54.610004858Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fg3EM4vVmVLMzqreKfIqADg","ver":1},"status":{"status":"write"}}},{"id":"1_1594621554.615406_417.1","section":"bucket.instance","name":"cngrzz-42:7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.1","timestamp":"2020-07-13T06:25:54.615406500Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fg3EM4vVmVLMzqreKfIqADg","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621554.617393_418.1","section":"bucket","name":"cngrzz-42","timestamp":"2020-07-13T06:25:54.617393527Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b8eCpo2wyeyoUHzcMY8mjUW","ver":1},"status":{"status":"write"}}},{"id":"1_1594621554.621183_419.1","section":"bucket","name":"cngrzz-42","timestamp":"2020-07-13T06:25:54.621183763Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b8eCpo2wyeyoUHzcMY8mjUW","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621554.669155_420.1","section":"bucket.instance","name":"cngrzz-43:7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.2","timestamp":"2020-07-13T06:25:54.669155022Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pruOl0R70hBg18xsfAzI4L_","ver":1},"status":{"status":"write"}}},{"id":"1_1594621554.673262_421.1","section":"bucket.instance","name":"cngrzz-43:7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.2","timestamp":"2020-07-13T06:25:54.673262872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pruOl0R70hBg18xsfAzI4L_","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621554.675281_422.1","section":"bucket","name":"cngrzz-43","timestamp":"2020-07-13T06:25:54.675281393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BQ_fGwBgvGz-aNmWMtyO2ml","ver":1},"status":{"status":"write"}}},{"id":"1_1594621554.679151_423.1","section":"bucket","name":"cngrzz-43","timestamp":"2020-07-13T06:25:54.679151359Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BQ_fGwBgvGz-aNmWMtyO2ml","ver":1},"status":{"status":"complete"}}}] 2020-07-13T06:27:07.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'c0ad3871-076a-4736-ba2c-6cf14ab7b93e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:07.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'c0ad3871-076a-4736-ba2c-6cf14ab7b93e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:07.255 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period c0ad3871-076a-4736-ba2c-6cf14ab7b93e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:07.296 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:07.296 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:07.546 INFO:teuthology.orchestra.run.smithi014.stdout:[{"id":"1_1594621554.610004_416.1","section":"bucket.instance","name":"cngrzz-42:7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.1","timestamp":"2020-07-13T06:25:54.610004858Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fg3EM4vVmVLMzqreKfIqADg","ver":1},"status":{"status":"write"}}},{"id":"1_1594621554.615406_417.1","section":"bucket.instance","name":"cngrzz-42:7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.1","timestamp":"2020-07-13T06:25:54.615406500Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fg3EM4vVmVLMzqreKfIqADg","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621554.617393_418.1","section":"bucket","name":"cngrzz-42","timestamp":"2020-07-13T06:25:54.617393527Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b8eCpo2wyeyoUHzcMY8mjUW","ver":1},"status":{"status":"write"}}},{"id":"1_1594621554.621183_419.1","section":"bucket","name":"cngrzz-42","timestamp":"2020-07-13T06:25:54.621183763Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b8eCpo2wyeyoUHzcMY8mjUW","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621554.669155_420.1","section":"bucket.instance","name":"cngrzz-43:7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.2","timestamp":"2020-07-13T06:25:54.669155022Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pruOl0R70hBg18xsfAzI4L_","ver":1},"status":{"status":"write"}}},{"id":"1_1594621554.673262_421.1","section":"bucket.instance","name":"cngrzz-43:7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.2","timestamp":"2020-07-13T06:25:54.673262872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pruOl0R70hBg18xsfAzI4L_","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621554.675281_422.1","section":"bucket","name":"cngrzz-43","timestamp":"2020-07-13T06:25:54.675281393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BQ_fGwBgvGz-aNmWMtyO2ml","ver":1},"status":{"status":"write"}}},{"id":"1_1594621554.679151_423.1","section":"bucket","name":"cngrzz-43","timestamp":"2020-07-13T06:25:54.679151359Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BQ_fGwBgvGz-aNmWMtyO2ml","ver":1},"status":{"status":"complete"}}}] 2020-07-13T06:27:07.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '9ca21b56-c204-42f4-9360-46419e3ebcc4', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:07.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '9ca21b56-c204-42f4-9360-46419e3ebcc4', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:07.548 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 9ca21b56-c204-42f4-9360-46419e3ebcc4 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:07.587 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:07.587 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:07.869 INFO:teuthology.orchestra.run.smithi017.stdout:[{"id":"1_1594621615.806672_268.1","section":"bucket.instance","name":"cngrzz-44:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.1","timestamp":"2020-07-13T06:26:55.806672456Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q5U-GWS9su9Eooqzqti-qpo","ver":1},"status":{"status":"write"}}},{"id":"1_1594621615.811591_269.1","section":"bucket.instance","name":"cngrzz-44:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.1","timestamp":"2020-07-13T06:26:55.811591104Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q5U-GWS9su9Eooqzqti-qpo","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621615.813548_270.1","section":"bucket","name":"cngrzz-44","timestamp":"2020-07-13T06:26:55.813548829Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AbK9okPRPUqflG5IcKI5--R","ver":1},"status":{"status":"write"}}},{"id":"1_1594621615.818363_271.1","section":"bucket","name":"cngrzz-44","timestamp":"2020-07-13T06:26:55.818363543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AbK9okPRPUqflG5IcKI5--R","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621615.873303_272.1","section":"bucket.instance","name":"cngrzz-45:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.2","timestamp":"2020-07-13T06:26:55.873303733Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7qE15voSviE5ulLuN_akQHV","ver":1},"status":{"status":"write"}}},{"id":"1_1594621615.877844_273.1","section":"bucket.instance","name":"cngrzz-45:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.2","timestamp":"2020-07-13T06:26:55.877844800Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7qE15voSviE5ulLuN_akQHV","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621615.879649_274.1","section":"bucket","name":"cngrzz-45","timestamp":"2020-07-13T06:26:55.879649783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lh8Eosj48J3sMI2euioGO5R","ver":1},"status":{"status":"write"}}},{"id":"1_1594621615.883650_275.1","section":"bucket","name":"cngrzz-45","timestamp":"2020-07-13T06:26:55.883650690Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lh8Eosj48J3sMI2euioGO5R","ver":1},"status":{"status":"complete"}}}] 2020-07-13T06:27:07.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '9ca21b56-c204-42f4-9360-46419e3ebcc4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:07.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '9ca21b56-c204-42f4-9360-46419e3ebcc4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:07.871 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 9ca21b56-c204-42f4-9360-46419e3ebcc4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:07.915 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:07.916 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:08.176 INFO:teuthology.orchestra.run.smithi014.stdout:[{"id":"1_1594621615.806672_268.1","section":"bucket.instance","name":"cngrzz-44:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.1","timestamp":"2020-07-13T06:26:55.806672456Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q5U-GWS9su9Eooqzqti-qpo","ver":1},"status":{"status":"write"}}},{"id":"1_1594621615.811591_269.1","section":"bucket.instance","name":"cngrzz-44:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.1","timestamp":"2020-07-13T06:26:55.811591104Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q5U-GWS9su9Eooqzqti-qpo","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621615.813548_270.1","section":"bucket","name":"cngrzz-44","timestamp":"2020-07-13T06:26:55.813548829Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AbK9okPRPUqflG5IcKI5--R","ver":1},"status":{"status":"write"}}},{"id":"1_1594621615.818363_271.1","section":"bucket","name":"cngrzz-44","timestamp":"2020-07-13T06:26:55.818363543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AbK9okPRPUqflG5IcKI5--R","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621615.873303_272.1","section":"bucket.instance","name":"cngrzz-45:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.2","timestamp":"2020-07-13T06:26:55.873303733Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7qE15voSviE5ulLuN_akQHV","ver":1},"status":{"status":"write"}}},{"id":"1_1594621615.877844_273.1","section":"bucket.instance","name":"cngrzz-45:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.2","timestamp":"2020-07-13T06:26:55.877844800Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7qE15voSviE5ulLuN_akQHV","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621615.879649_274.1","section":"bucket","name":"cngrzz-45","timestamp":"2020-07-13T06:26:55.879649783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lh8Eosj48J3sMI2euioGO5R","ver":1},"status":{"status":"write"}}},{"id":"1_1594621615.883650_275.1","section":"bucket","name":"cngrzz-45","timestamp":"2020-07-13T06:26:55.883650690Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lh8Eosj48J3sMI2euioGO5R","ver":1},"status":{"status":"complete"}}}] 2020-07-13T06:27:08.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '9ca21b56-c204-42f4-9360-46419e3ebcc4', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:08.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '9ca21b56-c204-42f4-9360-46419e3ebcc4', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:08.178 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 9ca21b56-c204-42f4-9360-46419e3ebcc4 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:08.218 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:08.218 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:08.485 INFO:teuthology.orchestra.run.smithi017.stdout:[{"id":"1_1594621615.806672_268.1","section":"bucket.instance","name":"cngrzz-44:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.1","timestamp":"2020-07-13T06:26:55.806672456Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q5U-GWS9su9Eooqzqti-qpo","ver":1},"status":{"status":"write"}}},{"id":"1_1594621615.811591_269.1","section":"bucket.instance","name":"cngrzz-44:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.1","timestamp":"2020-07-13T06:26:55.811591104Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q5U-GWS9su9Eooqzqti-qpo","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621615.813548_270.1","section":"bucket","name":"cngrzz-44","timestamp":"2020-07-13T06:26:55.813548829Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AbK9okPRPUqflG5IcKI5--R","ver":1},"status":{"status":"write"}}},{"id":"1_1594621615.818363_271.1","section":"bucket","name":"cngrzz-44","timestamp":"2020-07-13T06:26:55.818363543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AbK9okPRPUqflG5IcKI5--R","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621615.873303_272.1","section":"bucket.instance","name":"cngrzz-45:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.2","timestamp":"2020-07-13T06:26:55.873303733Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7qE15voSviE5ulLuN_akQHV","ver":1},"status":{"status":"write"}}},{"id":"1_1594621615.877844_273.1","section":"bucket.instance","name":"cngrzz-45:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.2","timestamp":"2020-07-13T06:26:55.877844800Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7qE15voSviE5ulLuN_akQHV","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621615.879649_274.1","section":"bucket","name":"cngrzz-45","timestamp":"2020-07-13T06:26:55.879649783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lh8Eosj48J3sMI2euioGO5R","ver":1},"status":{"status":"write"}}},{"id":"1_1594621615.883650_275.1","section":"bucket","name":"cngrzz-45","timestamp":"2020-07-13T06:26:55.883650690Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lh8Eosj48J3sMI2euioGO5R","ver":1},"status":{"status":"complete"}}}] 2020-07-13T06:27:08.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '9ca21b56-c204-42f4-9360-46419e3ebcc4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:08.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '9ca21b56-c204-42f4-9360-46419e3ebcc4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:08.488 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 9ca21b56-c204-42f4-9360-46419e3ebcc4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:08.528 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:08.529 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:08.769 INFO:teuthology.orchestra.run.smithi014.stdout:[{"id":"1_1594621615.806672_268.1","section":"bucket.instance","name":"cngrzz-44:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.1","timestamp":"2020-07-13T06:26:55.806672456Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q5U-GWS9su9Eooqzqti-qpo","ver":1},"status":{"status":"write"}}},{"id":"1_1594621615.811591_269.1","section":"bucket.instance","name":"cngrzz-44:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.1","timestamp":"2020-07-13T06:26:55.811591104Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q5U-GWS9su9Eooqzqti-qpo","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621615.813548_270.1","section":"bucket","name":"cngrzz-44","timestamp":"2020-07-13T06:26:55.813548829Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AbK9okPRPUqflG5IcKI5--R","ver":1},"status":{"status":"write"}}},{"id":"1_1594621615.818363_271.1","section":"bucket","name":"cngrzz-44","timestamp":"2020-07-13T06:26:55.818363543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AbK9okPRPUqflG5IcKI5--R","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621615.873303_272.1","section":"bucket.instance","name":"cngrzz-45:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.2","timestamp":"2020-07-13T06:26:55.873303733Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7qE15voSviE5ulLuN_akQHV","ver":1},"status":{"status":"write"}}},{"id":"1_1594621615.877844_273.1","section":"bucket.instance","name":"cngrzz-45:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.2","timestamp":"2020-07-13T06:26:55.877844800Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7qE15voSviE5ulLuN_akQHV","ver":1},"status":{"status":"complete"}}},{"id":"1_1594621615.879649_274.1","section":"bucket","name":"cngrzz-45","timestamp":"2020-07-13T06:26:55.879649783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lh8Eosj48J3sMI2euioGO5R","ver":1},"status":{"status":"write"}}},{"id":"1_1594621615.883650_275.1","section":"bucket","name":"cngrzz-45","timestamp":"2020-07-13T06:26:55.883650690Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lh8Eosj48J3sMI2euioGO5R","ver":1},"status":{"status":"complete"}}}] 2020-07-13T06:27:08.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:27:08.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:27:08.772 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:27:08.811 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:08.812 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:09.125 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:27:09.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:27:09.126 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:27:09.163 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:09.163 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:09.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:27:09.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:27:09.560 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:27:09.598 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:09.598 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:09.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:27:09.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:27:09.881 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:27:09.921 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:09.921 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:10.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '587bd98b-69d2-49ba-9dbd-2be72177aade', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:10.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '587bd98b-69d2-49ba-9dbd-2be72177aade', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:10.317 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 587bd98b-69d2-49ba-9dbd-2be72177aade --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:10.357 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:10.358 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:10.616 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2020-07-13T06:27:10.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '587bd98b-69d2-49ba-9dbd-2be72177aade', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:10.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '587bd98b-69d2-49ba-9dbd-2be72177aade', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:10.617 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 587bd98b-69d2-49ba-9dbd-2be72177aade --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:10.657 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:10.658 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:10.907 INFO:teuthology.orchestra.run.smithi014.stdout:[] 2020-07-13T06:27:10.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '587bd98b-69d2-49ba-9dbd-2be72177aade', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:10.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '587bd98b-69d2-49ba-9dbd-2be72177aade', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:10.909 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 587bd98b-69d2-49ba-9dbd-2be72177aade --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:10.947 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:10.948 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:11.209 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2020-07-13T06:27:11.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '587bd98b-69d2-49ba-9dbd-2be72177aade', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:11.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '587bd98b-69d2-49ba-9dbd-2be72177aade', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:11.211 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 587bd98b-69d2-49ba-9dbd-2be72177aade --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:11.252 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:11.253 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:11.509 INFO:teuthology.orchestra.run.smithi014.stdout:[] 2020-07-13T06:27:11.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'c0ad3871-076a-4736-ba2c-6cf14ab7b93e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:11.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'c0ad3871-076a-4736-ba2c-6cf14ab7b93e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:11.514 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period c0ad3871-076a-4736-ba2c-6cf14ab7b93e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:11.561 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:11.561 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:11.804 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2020-07-13T06:27:11.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'c0ad3871-076a-4736-ba2c-6cf14ab7b93e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:11.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'c0ad3871-076a-4736-ba2c-6cf14ab7b93e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:11.806 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period c0ad3871-076a-4736-ba2c-6cf14ab7b93e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:11.845 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:11.846 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:12.111 INFO:teuthology.orchestra.run.smithi014.stdout:[] 2020-07-13T06:27:12.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'c0ad3871-076a-4736-ba2c-6cf14ab7b93e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:12.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'c0ad3871-076a-4736-ba2c-6cf14ab7b93e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:12.113 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period c0ad3871-076a-4736-ba2c-6cf14ab7b93e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:12.152 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:12.152 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:12.409 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2020-07-13T06:27:12.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'c0ad3871-076a-4736-ba2c-6cf14ab7b93e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:12.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'c0ad3871-076a-4736-ba2c-6cf14ab7b93e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:12.411 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period c0ad3871-076a-4736-ba2c-6cf14ab7b93e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:12.450 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:12.451 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:12.712 INFO:teuthology.orchestra.run.smithi014.stdout:[] 2020-07-13T06:27:12.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '9ca21b56-c204-42f4-9360-46419e3ebcc4', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:12.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '9ca21b56-c204-42f4-9360-46419e3ebcc4', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:12.714 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 9ca21b56-c204-42f4-9360-46419e3ebcc4 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:12.755 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:12.755 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:13.022 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2020-07-13T06:27:13.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '9ca21b56-c204-42f4-9360-46419e3ebcc4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:13.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '9ca21b56-c204-42f4-9360-46419e3ebcc4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:13.024 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 9ca21b56-c204-42f4-9360-46419e3ebcc4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:13.064 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:13.065 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:13.325 INFO:teuthology.orchestra.run.smithi014.stdout:[] 2020-07-13T06:27:13.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '9ca21b56-c204-42f4-9360-46419e3ebcc4', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:13.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '9ca21b56-c204-42f4-9360-46419e3ebcc4', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:13.327 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 9ca21b56-c204-42f4-9360-46419e3ebcc4 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:13.367 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:13.367 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:13.620 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2020-07-13T06:27:13.621 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '9ca21b56-c204-42f4-9360-46419e3ebcc4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:13.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '9ca21b56-c204-42f4-9360-46419e3ebcc4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:13.622 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 9ca21b56-c204-42f4-9360-46419e3ebcc4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:13.660 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:13.660 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:13.911 INFO:teuthology.orchestra.run.smithi014.stdout:[] 2020-07-13T06:27:13.912 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2020-07-13T06:27:13.914 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-46 2020-07-13T06:27:14.174 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-47 2020-07-13T06:27:15.242 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-48 2020-07-13T06:27:16.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:16.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:16.284 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:16.326 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:16.326 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:16.554 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621203.033096_83.1","last_update":"2020-07-13T06:20:03.033096Z"},{"marker":"1_1594621486.213353_328.1","last_update":"2020-07-13T06:24:46.213353Z"},{"marker":"1_1594621137.230195_96.1","last_update":"2020-07-13T06:18:57.230195Z"},{"marker":"1_1594621636.457941_396.1","last_update":"2020-07-13T06:27:16.457941Z"}] 2020-07-13T06:27:16.555 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-13T06:27:16.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:16.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:16.556 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:16.598 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:16.598 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:16.800 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15669168566042326328},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621203.033096_83.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:20:03.033096Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.457941_396.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.457941Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:27:16.801 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-13T06:27:16.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:16.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:16.803 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:16.870 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:16.870 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:17.125 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621203.033096_83.1","last_update":"2020-07-13T06:20:03.033096Z"},{"marker":"1_1594621486.213353_328.1","last_update":"2020-07-13T06:24:46.213353Z"},{"marker":"1_1594621137.230195_96.1","last_update":"2020-07-13T06:18:57.230195Z"},{"marker":"1_1594621636.558054_398.1","last_update":"2020-07-13T06:27:16.558054Z"}] 2020-07-13T06:27:17.125 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-13T06:27:17.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:17.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:17.127 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:17.224 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:17.224 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:17.472 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2389703693262437130},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621203.033096_83.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:20:03.033096Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:27:17.473 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-13T06:27:17.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:17.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:17.474 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:17.556 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:17.557 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:17.785 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621203.033096_83.1","last_update":"2020-07-13T06:20:03.033096Z"},{"marker":"1_1594621486.213353_328.1","last_update":"2020-07-13T06:24:46.213353Z"},{"marker":"1_1594621137.230195_96.1","last_update":"2020-07-13T06:18:57.230195Z"},{"marker":"1_1594621636.558054_398.1","last_update":"2020-07-13T06:27:16.558054Z"}] 2020-07-13T06:27:17.785 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-13T06:27:17.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:17.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:17.789 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:17.833 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:17.833 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:18.104 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12943611757838484443},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621203.033096_83.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:20:03.033096Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:27:18.105 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-13T06:27:18.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:18.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:18.107 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:18.186 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:18.187 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:18.432 INFO:teuthology.orchestra.run.smithi014.stdout:[{"marker":"1_1594621203.127419_61.1","last_update":"2020-07-13T06:20:03.127419Z"},{"marker":"1_1594621486.218770_296.1","last_update":"2020-07-13T06:24:46.218770Z"},{"marker":"1_1594621137.114488_103.3","last_update":"2020-07-13T06:18:57.114488Z"},{"marker":"1_1594621636.554937_308.1","last_update":"2020-07-13T06:27:16.554937Z"}] 2020-07-13T06:27:18.432 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-13T06:27:18.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:18.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:18.434 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:18.477 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:18.478 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:18.710 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6157611387314156630},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621203.127419_61.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:20:03.127419Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.218770_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.218770Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.114488_103.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.114488Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.554937_308.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.554937Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:27:18.711 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-13T06:27:18.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:18.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:18.712 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:18.753 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:18.753 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:19.009 INFO:teuthology.orchestra.run.smithi014.stdout:[{"marker":"1_1594621203.127419_61.1","last_update":"2020-07-13T06:20:03.127419Z"},{"marker":"1_1594621486.218770_296.1","last_update":"2020-07-13T06:24:46.218770Z"},{"marker":"1_1594621137.114488_103.3","last_update":"2020-07-13T06:18:57.114488Z"},{"marker":"1_1594621636.554937_308.1","last_update":"2020-07-13T06:27:16.554937Z"}] 2020-07-13T06:27:19.009 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-13T06:27:19.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:19.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:19.011 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:19.055 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:19.056 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:19.299 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3188359685164663541},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621203.127419_61.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:20:03.127419Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.218770_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.218770Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.114488_103.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.114488Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.554937_308.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.554937Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:27:19.300 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-13T06:27:19.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:19.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:19.302 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:19.392 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:19.392 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:19.630 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621202.985988_64.1","last_update":"2020-07-13T06:20:02.985988Z"},{"marker":"1_1594621485.914168_264.1","last_update":"2020-07-13T06:24:45.914168Z"},{"marker":"1_1594621137.234104_70.1","last_update":"2020-07-13T06:18:57.234104Z"},{"marker":"1_1594621636.463177_329.1","last_update":"2020-07-13T06:27:16.463177Z"}] 2020-07-13T06:27:19.630 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-13T06:27:19.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:19.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:19.632 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:19.711 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:19.712 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:19.971 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3540715737640754362},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621202.985988_64.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:20:02.985988Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621485.914168_264.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:45.914168Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.234104_70.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.234104Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.463177_329.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.463177Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:27:19.971 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-13T06:27:19.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:19.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:19.973 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:20.058 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:20.059 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:20.296 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621202.985988_64.1","last_update":"2020-07-13T06:20:02.985988Z"},{"marker":"1_1594621485.914168_264.1","last_update":"2020-07-13T06:24:45.914168Z"},{"marker":"1_1594621137.234104_70.1","last_update":"2020-07-13T06:18:57.234104Z"},{"marker":"1_1594621636.463177_329.1","last_update":"2020-07-13T06:27:16.463177Z"}] 2020-07-13T06:27:20.297 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-13T06:27:20.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:20.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:20.299 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:20.355 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:20.355 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:20.600 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11134568813707895163},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621202.985988_64.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:20:02.985988Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621485.914168_264.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:45.914168Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.234104_70.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.234104Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.463177_329.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.463177Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:27:20.601 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-13T06:27:20.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:27:20.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:27:20.602 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:27:20.635 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:20.636 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:20.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:20.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:20.893 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:20.977 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:20.977 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:21.234 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2020-07-13T06:27:21.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:27:21.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:27:21.236 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:27:21.272 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:21.272 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:21.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:21.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:21.547 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:21.625 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:21.625 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:21.873 INFO:teuthology.orchestra.run.smithi014.stdout:[] 2020-07-13T06:27:21.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:27:21.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:27:21.875 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:27:21.912 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:21.912 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:22.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:22.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:27:22.165 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:27:22.243 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:22.243 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:22.507 INFO:teuthology.orchestra.run.smithi017.stdout:[] 2020-07-13T06:27:22.508 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ok 2020-07-13T06:27:22.509 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:27:22.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:27:22.510 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:27:22.554 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:22.555 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:22.578 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:27:22.572+0000 7f304d9e00c0 0 failed reading obj info from .rgw.root:zone_info.5fd641b5-9375-4754-8474-97ebd79eafb6: (2) No such file or directory 2020-07-13T06:27:22.579 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:27:22.572+0000 7f304d9e00c0 0 WARNING: could not read zone params for zone id=5fd641b5-9375-4754-8474-97ebd79eafb6 name=test-zone1 2020-07-13T06:27:22.580 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:27:22.572+0000 7f304d9e00c0 0 failed reading obj info from .rgw.root:zone_info.e66a1e9b-6861-42ff-8eed-7287194f3fe0: (2) No such file or directory 2020-07-13T06:27:22.580 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:27:22.572+0000 7f304d9e00c0 0 WARNING: could not read zone params for zone id=e66a1e9b-6861-42ff-8eed-7287194f3fe0 name=test-zone3 2020-07-13T06:27:22.589 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"HVCB2URARHK5PDP7","secret_key":"iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668"} 2020-07-13T06:27:22.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:27:22.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:27:22.592 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:27:22.669 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:27:22.670 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:27:22.922 INFO:teuthology.orchestra.run.smithi014.stderr:Sending period to new master zone 5fd641b5-9375-4754-8474-97ebd79eafb6 2020-07-13T06:27:23.018 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"9ca21b56-c204-42f4-9360-46419e3ebcc4","epoch":2,"predecessor_uuid":"c0ad3871-076a-4736-ba2c-6cf14ab7b93e","sync_status":["1_1594621554.679151_423.1","","",""],"period_map":{"id":"9ca21b56-c204-42f4-9360-46419e3ebcc4","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","zones":[{"id":"5fd641b5-9375-4754-8474-97ebd79eafb6","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","endpoints":["http://smithi014.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":""},{"id":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","name":"test-zone4","endpoints":["http://smithi014.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5fd641b5-9375-4754-8474-97ebd79eafb6","val":1497020214},{"key":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","val":508986481},{"key":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","val":2882337890},{"key":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","val":617933293}]},"master_zonegroup":"test-zonegroup","master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","realm_name":"test-realm","realm_epoch":4} 2020-07-13T06:27:23.019 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2020-07-13T06:28:23.020 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-49 2020-07-13T06:28:23.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:28:23.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:28:23.140 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:28:23.178 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:28:23.178 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:28:23.393 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=9ca21b56-c204-42f4-9360-46419e3ebcc4 2020-07-13T06:28:23.432 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621703.066777_321.1","last_update":"2020-07-13T06:28:23.066777Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:28:23.433 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:28:23.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:28:23.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:28:23.435 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:28:23.476 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:28:23.477 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:28:23.715 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621635.513825_291.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:27:15.513825Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:28:23.716 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594621703.066777_321.1 target=1_1594621635.513825_291.1 2020-07-13T06:28:28.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:28:28.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:28:28.719 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:28:28.767 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:28:28.767 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:28:29.029 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621635.513825_291.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:27:15.513825Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:28:29.029 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594621703.066777_321.1 target=1_1594621635.513825_291.1 2020-07-13T06:28:34.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:28:34.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:28:34.033 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:28:34.074 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:28:34.075 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:28:34.322 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621635.513825_291.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:27:15.513825Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:28:34.323 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594621703.066777_321.1 target=1_1594621635.513825_291.1 2020-07-13T06:28:39.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:28:39.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:28:39.327 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:28:39.368 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:28:39.368 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:28:39.614 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621635.513825_291.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:27:15.513825Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:28:39.614 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594621703.066777_321.1 target=1_1594621635.513825_291.1 2020-07-13T06:28:44.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:28:44.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:28:44.618 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:28:44.659 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:28:44.659 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:28:44.897 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621635.513825_291.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:27:15.513825Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:28:44.898 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594621703.066777_321.1 target=1_1594621635.513825_291.1 2020-07-13T06:28:49.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:28:49.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:28:49.901 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:28:49.941 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:28:49.941 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:28:50.166 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621635.513825_291.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:27:15.513825Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:28:50.167 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594621703.066777_321.1 target=1_1594621635.513825_291.1 2020-07-13T06:28:55.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:28:55.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:28:55.171 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:28:55.215 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:28:55.215 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:28:55.459 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621635.513825_291.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:27:15.513825Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:28:55.460 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594621703.066777_321.1 target=1_1594621635.513825_291.1 2020-07-13T06:29:00.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:29:00.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:29:00.465 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:29:00.509 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:29:00.509 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:29:00.755 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621635.513825_291.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:27:15.513825Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:29:00.756 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594621703.066777_321.1 target=1_1594621635.513825_291.1 2020-07-13T06:29:05.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:29:05.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:29:05.759 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:29:05.802 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:29:05.802 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:29:06.045 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621635.513825_291.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:27:15.513825Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:29:06.046 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594621703.066777_321.1 target=1_1594621635.513825_291.1 2020-07-13T06:29:11.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:29:11.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:29:11.053 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:29:11.106 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:29:11.106 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:29:11.357 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621635.513825_291.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:27:15.513825Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:29:11.358 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594621703.066777_321.1 target=1_1594621635.513825_291.1 2020-07-13T06:29:16.360 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:29:16.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:29:16.362 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:29:16.409 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:29:16.409 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:29:16.648 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621635.513825_291.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:27:15.513825Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:29:16.649 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594621703.066777_321.1 target=1_1594621635.513825_291.1 2020-07-13T06:29:21.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:29:21.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:29:21.652 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:29:21.694 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:29:21.694 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:29:21.933 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621635.513825_291.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:27:15.513825Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:29:21.935 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594621703.066777_321.1 target=1_1594621635.513825_291.1 2020-07-13T06:29:26.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:29:26.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:29:26.938 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:29:26.981 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:29:26.981 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:29:27.233 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621703.066777_321.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:28:23.066777Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:29:27.233 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:29:27.234 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:29:27.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:29:27.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:29:27.236 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:29:27.276 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:29:27.277 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:29:27.521 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621703.066777_321.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:28:23.066777Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:29:27.521 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:29:27.522 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:29:27.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:29:27.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:29:27.524 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:29:27.565 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:29:27.565 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:29:27.828 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621703.066777_321.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:28:23.066777Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:29:27.828 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:29:27.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=5fd641b5-9375-4754-8474-97ebd79eafb6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:29:27.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=5fd641b5-9375-4754-8474-97ebd79eafb6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:29:27.856 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=5fd641b5-9375-4754-8474-97ebd79eafb6 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:29:27.913 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:29:27.913 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:29:27.943 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:29:27.941+0000 7f8bad2260c0 0 failed reading obj info from .rgw.root:zone_info.5fd641b5-9375-4754-8474-97ebd79eafb6: (2) No such file or directory 2020-07-13T06:29:27.944 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:29:27.941+0000 7f8bad2260c0 0 WARNING: could not read zone params for zone id=5fd641b5-9375-4754-8474-97ebd79eafb6 name=test-zone1 2020-07-13T06:29:27.946 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:29:27.941+0000 7f8bad2260c0 0 failed reading obj info from .rgw.root:zone_info.e66a1e9b-6861-42ff-8eed-7287194f3fe0: (2) No such file or directory 2020-07-13T06:29:27.946 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:29:27.941+0000 7f8bad2260c0 0 WARNING: could not read zone params for zone id=e66a1e9b-6861-42ff-8eed-7287194f3fe0 name=test-zone3 2020-07-13T06:29:27.959 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"HVCB2URARHK5PDP7","secret_key":"iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668"} 2020-07-13T06:29:27.960 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:29:27.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:29:27.962 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:29:28.040 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:29:28.040 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:29:28.283 INFO:teuthology.orchestra.run.smithi014.stderr:Sending period to new master zone 5fd641b5-9375-4754-8474-97ebd79eafb6 2020-07-13T06:29:28.379 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"9ca21b56-c204-42f4-9360-46419e3ebcc4","epoch":3,"predecessor_uuid":"c0ad3871-076a-4736-ba2c-6cf14ab7b93e","sync_status":["1_1594621554.679151_423.1","","",""],"period_map":{"id":"9ca21b56-c204-42f4-9360-46419e3ebcc4","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","zones":[{"id":"5fd641b5-9375-4754-8474-97ebd79eafb6","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","endpoints":["http://smithi014.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6"},{"id":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","name":"test-zone4","endpoints":["http://smithi014.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5fd641b5-9375-4754-8474-97ebd79eafb6","val":1497020214},{"key":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","val":508986481},{"key":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","val":2882337890},{"key":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","val":617933293}]},"master_zonegroup":"test-zonegroup","master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","realm_name":"test-realm","realm_epoch":4} 2020-07-13T06:29:28.380 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "5fd641b5-9375-4754-8474-97ebd79eafb6" 2020-07-13T06:30:28.668 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:30:28.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:30:28.672 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:30:28.713 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:30:28.714 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:30:28.743 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:30:28.740+0000 7f3f7d48a0c0 0 failed reading obj info from .rgw.root:zone_info.5fd641b5-9375-4754-8474-97ebd79eafb6: (2) No such file or directory 2020-07-13T06:30:28.744 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:30:28.740+0000 7f3f7d48a0c0 0 WARNING: could not read zone params for zone id=5fd641b5-9375-4754-8474-97ebd79eafb6 name=test-zone1 2020-07-13T06:30:28.748 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:30:28.744+0000 7f3f7d48a0c0 0 failed reading obj info from .rgw.root:zone_info.e66a1e9b-6861-42ff-8eed-7287194f3fe0: (2) No such file or directory 2020-07-13T06:30:28.749 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:30:28.744+0000 7f3f7d48a0c0 0 WARNING: could not read zone params for zone id=e66a1e9b-6861-42ff-8eed-7287194f3fe0 name=test-zone3 2020-07-13T06:30:28.763 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"HVCB2URARHK5PDP7","secret_key":"iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668"} 2020-07-13T06:30:28.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:30:28.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:30:28.766 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:30:28.850 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:30:28.851 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:30:29.097 INFO:teuthology.orchestra.run.smithi014.stderr:Sending period to new master zone 5fd641b5-9375-4754-8474-97ebd79eafb6 2020-07-13T06:30:29.198 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"9ca21b56-c204-42f4-9360-46419e3ebcc4","epoch":4,"predecessor_uuid":"c0ad3871-076a-4736-ba2c-6cf14ab7b93e","sync_status":["1_1594621554.679151_423.1","","",""],"period_map":{"id":"9ca21b56-c204-42f4-9360-46419e3ebcc4","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","zones":[{"id":"5fd641b5-9375-4754-8474-97ebd79eafb6","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","endpoints":["http://smithi014.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6"},{"id":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","name":"test-zone4","endpoints":["http://smithi014.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5fd641b5-9375-4754-8474-97ebd79eafb6","val":1497020214},{"key":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","val":508986481},{"key":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","val":2882337890},{"key":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","val":617933293}]},"master_zonegroup":"test-zonegroup","master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","realm_name":"test-realm","realm_epoch":4} 2020-07-13T06:30:29.199 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2020-07-13T06:31:29.201 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:31:29.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:31:29.205 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:31:29.258 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:31:29.259 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:31:29.276 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:31:29.271+0000 7fdb11f3e0c0 0 failed reading obj info from .rgw.root:zone_info.5fd641b5-9375-4754-8474-97ebd79eafb6: (2) No such file or directory 2020-07-13T06:31:29.277 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:31:29.271+0000 7fdb11f3e0c0 0 WARNING: could not read zone params for zone id=5fd641b5-9375-4754-8474-97ebd79eafb6 name=test-zone1 2020-07-13T06:31:29.281 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:31:29.275+0000 7fdb11f3e0c0 0 failed reading obj info from .rgw.root:zone_info.e66a1e9b-6861-42ff-8eed-7287194f3fe0: (2) No such file or directory 2020-07-13T06:31:29.281 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:31:29.275+0000 7fdb11f3e0c0 0 WARNING: could not read zone params for zone id=e66a1e9b-6861-42ff-8eed-7287194f3fe0 name=test-zone3 2020-07-13T06:31:29.294 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"HVCB2URARHK5PDP7","secret_key":"iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668"} 2020-07-13T06:31:29.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:31:29.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:31:29.297 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:31:29.379 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:31:29.380 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:31:29.613 INFO:teuthology.orchestra.run.smithi014.stderr:Sending period to new master zone 5fd641b5-9375-4754-8474-97ebd79eafb6 2020-07-13T06:31:29.705 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"9ca21b56-c204-42f4-9360-46419e3ebcc4","epoch":5,"predecessor_uuid":"c0ad3871-076a-4736-ba2c-6cf14ab7b93e","sync_status":["1_1594621554.679151_423.1","","",""],"period_map":{"id":"9ca21b56-c204-42f4-9360-46419e3ebcc4","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","zones":[{"id":"5fd641b5-9375-4754-8474-97ebd79eafb6","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","endpoints":["http://smithi014.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","name":"test-zone4","endpoints":["http://smithi014.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5fd641b5-9375-4754-8474-97ebd79eafb6","val":1497020214},{"key":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","val":508986481},{"key":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","val":2882337890},{"key":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","val":617933293}]},"master_zonegroup":"test-zonegroup","master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","realm_name":"test-realm","realm_epoch":4} 2020-07-13T06:31:29.706 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2020-07-13T06:32:29.707 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ok 2020-07-13T06:32:29.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:29.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:29.713 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:32:29.762 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:29.763 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:29.976 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=9ca21b56-c204-42f4-9360-46419e3ebcc4 2020-07-13T06:32:30.019 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621703.066777_321.1","last_update":"2020-07-13T06:28:23.066777Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:32:30.020 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:32:30.020 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:30.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:30.021 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:32:30.061 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:30.061 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:30.304 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621703.066777_321.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:28:23.066777Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:32:30.304 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:32:30.305 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:32:30.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:30.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:30.307 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:32:30.351 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:30.351 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:30.597 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621703.066777_321.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:28:23.066777Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:32:30.598 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:32:30.599 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:32:30.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:30.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:30.601 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:32:30.640 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:30.641 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:31.010 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594621703.066777_321.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:28:23.066777Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:32:31.011 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:32:31.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HVCB2URARHK5PDP7', '--secret', 'iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:31.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HVCB2URARHK5PDP7', '--secret', 'iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:31.014 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key HVCB2URARHK5PDP7 --secret iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:32:31.092 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:31.092 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:31.152 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:32:31.149+0000 7f247bd2f0c0 0 failed reading obj info from .rgw.root:zone_info.5fd641b5-9375-4754-8474-97ebd79eafb6: (2) No such file or directory 2020-07-13T06:32:31.153 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:32:31.149+0000 7f247bd2f0c0 0 WARNING: could not read zone params for zone id=5fd641b5-9375-4754-8474-97ebd79eafb6 name=test-zone1 2020-07-13T06:32:31.155 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:32:31.153+0000 7f247bd2f0c0 0 failed reading obj info from .rgw.root:zone_info.e66a1e9b-6861-42ff-8eed-7287194f3fe0: (2) No such file or directory 2020-07-13T06:32:31.155 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:32:31.153+0000 7f247bd2f0c0 0 WARNING: could not read zone params for zone id=e66a1e9b-6861-42ff-8eed-7287194f3fe0 name=test-zone3 2020-07-13T06:32:31.163 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"8d8ae01f-d4d3-4e6a-95d6-592bfb13118c","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"HVCB2URARHK5PDP7","secret_key":"iYV2KEF2qwDU8wm60aXeY8TNpqAnv4Eh"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668"} 2020-07-13T06:32:31.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:31.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:31.165 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:32:31.244 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:31.244 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:38.985 INFO:teuthology.orchestra.run.smithi014.stderr:Sending period to new master zone 5fd641b5-9375-4754-8474-97ebd79eafb6 2020-07-13T06:32:39.082 INFO:teuthology.orchestra.run.smithi014.stdout:{"id":"9ca21b56-c204-42f4-9360-46419e3ebcc4","epoch":6,"predecessor_uuid":"c0ad3871-076a-4736-ba2c-6cf14ab7b93e","sync_status":["1_1594621554.679151_423.1","","",""],"period_map":{"id":"9ca21b56-c204-42f4-9360-46419e3ebcc4","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","zones":[{"id":"5fd641b5-9375-4754-8474-97ebd79eafb6","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","endpoints":["http://smithi014.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"8d8ae01f-d4d3-4e6a-95d6-592bfb13118c","name":"remove","endpoints":[],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","name":"test-zone4","endpoints":["http://smithi014.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5fd641b5-9375-4754-8474-97ebd79eafb6","val":1497020214},{"key":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","val":508986481},{"key":"8d8ae01f-d4d3-4e6a-95d6-592bfb13118c","val":2566936095},{"key":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","val":2882337890},{"key":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","val":617933293}]},"master_zonegroup":"test-zonegroup","master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","realm_name":"test-realm","realm_epoch":4} 2020-07-13T06:32:39.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:39.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:39.084 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:32:39.122 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:39.122 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:39.144 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:32:39.138+0000 7f948022b0c0 0 failed reading obj info from .rgw.root:zone_info.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e: (2) No such file or directory 2020-07-13T06:32:39.144 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:32:39.138+0000 7f948022b0c0 0 WARNING: could not read zone params for zone id=7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e name=test-zone2 2020-07-13T06:32:39.145 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:32:39.138+0000 7f948022b0c0 0 failed reading obj info from .rgw.root:zone_info.d4145d87-2321-4625-b4d5-2fd46dfc82ef: (2) No such file or directory 2020-07-13T06:32:39.145 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:32:39.138+0000 7f948022b0c0 0 WARNING: could not read zone params for zone id=d4145d87-2321-4625-b4d5-2fd46dfc82ef name=test-zone4 2020-07-13T06:32:39.155 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","zones":[{"id":"5fd641b5-9375-4754-8474-97ebd79eafb6","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","endpoints":["http://smithi014.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","name":"test-zone4","endpoints":["http://smithi014.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","sync_policy":{"groups":[]}} 2020-07-13T06:32:39.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:39.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:39.157 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:32:39.243 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:39.243 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:39.263 INFO:teuthology.orchestra.run.smithi017.stderr:zone name remove not found in zonegroup test-zonegroup 2020-07-13T06:32:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-07-13T06:32:39.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:39.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:39.273 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-13T06:32:39.315 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:39.315 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:39.357 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:32:39.353+0000 7f173e2bc0c0 0 failed reading obj info from .rgw.root:zone_info.5fd641b5-9375-4754-8474-97ebd79eafb6: (2) No such file or directory 2020-07-13T06:32:39.357 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:32:39.353+0000 7f173e2bc0c0 0 WARNING: could not read zone params for zone id=5fd641b5-9375-4754-8474-97ebd79eafb6 name=test-zone1 2020-07-13T06:32:39.359 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:32:39.357+0000 7f173e2bc0c0 0 failed reading obj info from .rgw.root:zone_info.e66a1e9b-6861-42ff-8eed-7287194f3fe0: (2) No such file or directory 2020-07-13T06:32:39.359 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:32:39.357+0000 7f173e2bc0c0 0 WARNING: could not read zone params for zone id=e66a1e9b-6861-42ff-8eed-7287194f3fe0 name=test-zone3 2020-07-13T06:32:39.364 INFO:teuthology.orchestra.run.smithi014.stderr:2020-07-13T06:32:39.361+0000 7f173e2bc0c0 0 zone id 8d8ae01f-d4d3-4e6a-95d6-592bfb13118c is not a part of zonegroup default 2020-07-13T06:32:39.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:39.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:39.381 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:32:39.420 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:39.420 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:39.482 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:32:39.474+0000 7fab81aec0c0 0 WARNING: can't generate connection for zone 8d8ae01f-d4d3-4e6a-95d6-592bfb13118c id remove: no endpoints defined 2020-07-13T06:32:39.769 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"9ca21b56-c204-42f4-9360-46419e3ebcc4","epoch":7,"predecessor_uuid":"c0ad3871-076a-4736-ba2c-6cf14ab7b93e","sync_status":["1_1594621554.679151_423.1","","",""],"period_map":{"id":"9ca21b56-c204-42f4-9360-46419e3ebcc4","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","zones":[{"id":"5fd641b5-9375-4754-8474-97ebd79eafb6","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","endpoints":["http://smithi014.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","name":"test-zone4","endpoints":["http://smithi014.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5fd641b5-9375-4754-8474-97ebd79eafb6","val":1497020214},{"key":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","val":508986481},{"key":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","val":2882337890},{"key":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","val":617933293}]},"master_zonegroup":"test-zonegroup","master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","realm_name":"test-realm","realm_epoch":4} 2020-07-13T06:32:39.770 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2020-07-13T06:32:39.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:39.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:39.772 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:32:39.846 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:39.847 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:39.882 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"1cf6c64f-6c91-41cc-bebb-eb6834aa0e15","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","sync_policy":{"groups":[]}} 2020-07-13T06:32:39.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:39.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:39.884 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:32:39.962 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:39.962 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:40.023 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"5df2c765-e665-47e9-a9e8-9b06679fd46b","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668"} 2020-07-13T06:32:40.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:40.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:40.028 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:32:40.112 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:40.113 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:40.428 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"9ca21b56-c204-42f4-9360-46419e3ebcc4","epoch":8,"predecessor_uuid":"c0ad3871-076a-4736-ba2c-6cf14ab7b93e","sync_status":["1_1594621554.679151_423.1","","",""],"period_map":{"id":"9ca21b56-c204-42f4-9360-46419e3ebcc4","zonegroups":[{"id":"1cf6c64f-6c91-41cc-bebb-eb6834aa0e15","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"5df2c765-e665-47e9-a9e8-9b06679fd46b","zones":[{"id":"5df2c765-e665-47e9-a9e8-9b06679fd46b","name":"remove","endpoints":[],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","sync_policy":{"groups":[]}},{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","zones":[{"id":"5fd641b5-9375-4754-8474-97ebd79eafb6","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","endpoints":["http://smithi014.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","name":"test-zone4","endpoints":["http://smithi014.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5df2c765-e665-47e9-a9e8-9b06679fd46b","val":2418317855},{"key":"5fd641b5-9375-4754-8474-97ebd79eafb6","val":1497020214},{"key":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","val":508986481},{"key":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","val":2882337890},{"key":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","val":617933293}]},"master_zonegroup":"test-zonegroup","master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","realm_name":"test-realm","realm_epoch":4} 2020-07-13T06:32:40.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:40.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:40.430 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:32:40.513 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:40.513 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:40.564 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:32:40.558+0000 7ff3038e90c0 0 zone id 5df2c765-e665-47e9-a9e8-9b06679fd46b is not a part of zonegroup test-zonegroup 2020-07-13T06:32:40.565 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:32:40.558+0000 7ff3038e90c0 0 zone id 5df2c765-e665-47e9-a9e8-9b06679fd46b is not a part of zonegroup default 2020-07-13T06:32:40.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:40.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:40.594 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:32:40.676 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:40.677 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:40.723 INFO:teuthology.orchestra.run.smithi017.stderr:2020-07-13T06:32:40.714+0000 7f39bd3560c0 0 ERROR: zonegroup remove_zg has a non existent master zone 2020-07-13T06:32:40.728 DEBUG:teuthology.orchestra.run:got remote process result: 22 2020-07-13T06:32:40.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:40.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:40.730 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:32:40.812 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:40.812 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:40.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:40.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:40.848 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:32:40.932 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:40.932 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:41.226 INFO:teuthology.orchestra.run.smithi017.stdout:{"id":"9ca21b56-c204-42f4-9360-46419e3ebcc4","epoch":9,"predecessor_uuid":"c0ad3871-076a-4736-ba2c-6cf14ab7b93e","sync_status":["1_1594621554.679151_423.1","","",""],"period_map":{"id":"9ca21b56-c204-42f4-9360-46419e3ebcc4","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","zones":[{"id":"5fd641b5-9375-4754-8474-97ebd79eafb6","name":"test-zone1","endpoints":["http://smithi017.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","name":"test-zone2","endpoints":["http://smithi014.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","name":"test-zone4","endpoints":["http://smithi014.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","name":"test-zone3","endpoints":["http://smithi017.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5fd641b5-9375-4754-8474-97ebd79eafb6","val":1497020214},{"key":"7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","val":508986481},{"key":"d4145d87-2321-4625-b4d5-2fd46dfc82ef","val":2882337890},{"key":"e66a1e9b-6861-42ff-8eed-7287194f3fe0","val":617933293}]},"master_zonegroup":"test-zonegroup","master_zone":"5fd641b5-9375-4754-8474-97ebd79eafb6","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"3bffea65-3664-4393-a4a6-65c46ad5d668","realm_name":"test-realm","realm_epoch":4} 2020-07-13T06:32:41.227 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2020-07-13T06:32:41.228 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-50 2020-07-13T06:32:41.836 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-51 2020-07-13T06:32:42.006 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-52 2020-07-13T06:32:42.080 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-53 2020-07-13T06:32:42.125 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-54 2020-07-13T06:32:42.198 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-55 2020-07-13T06:32:45.050 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2020-07-13T06:32:45.051 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-56 2020-07-13T06:32:45.823 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-57 2020-07-13T06:32:46.461 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-58 2020-07-13T06:32:47.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'cngrzz-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:47.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'cngrzz-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:47.362 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket cngrzz-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:32:47.407 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:47.408 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:47.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'cngrzz-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:47.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'cngrzz-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:47.741 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket cngrzz-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:32:47.822 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:47.823 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:48.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'cngrzz-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:48.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'cngrzz-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:32:48.167 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket cngrzz-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:32:48.248 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:48.249 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:48.586 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:48.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:48.587 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:32:48.668 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:48.668 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:48.922 INFO:teuthology.orchestra.run.smithi017.stdout:["bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:1","obj_delete_at_hint.0000000078","meta.history","2020-07-13-06-7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.2-cngrzz-43","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33:8","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30:8","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:3","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:3","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.17-cngrzz-8","obj_delete_at_hint.0000000028","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31:8","obj_delete_at_hint.0000000069","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-13:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.22:1","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7772.4-cngrzz-53","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:3","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.47-cngrzz-38","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26-cngrzz-17","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-37:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.46:10","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33-cngrzz-24","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31:8","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.39-cngrzz-30","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:8","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7772.1-cngrzz-50","obj_delete_at_hint.0000000024","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:1","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.6-cngrzz-6","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7240.2-cngrzz-45","data_log.3","datalog.sync-status.shard.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.0","datalog.sync-status.shard.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.2","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-49:5fd641b5-9375-4754-8474-97ebd79eafb6.7539.1:5","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.50-cngrzz-41","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31-cngrzz-22","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:1","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:7","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.45-cngrzz-36","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7240.5-cngrzz-48","datalog.sync-status.shard.e66a1e9b-6861-42ff-8eed-7287194f3fe0.2","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-38:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.47:10","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7772.3-cngrzz-52","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7240.3-cngrzz-46","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.48-cngrzz-39","obj_delete_at_hint.0000000110","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19-cngrzz-10","datalog.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0","obj_delete_at_hint.0000000096","data.full-sync.index.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.1","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7772.6-cngrzz-55","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35:8","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.2-cngrzz-2","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.4-cngrzz-4","obj_delete_at_hint.0000000006","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28:8","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7772.8-cngrzz-57","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35:8","obj_delete_at_hint.0000000022","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.5-cngrzz-5","bilog.trim","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:3","obj_delete_at_hint.0000000016","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7772.5-cngrzz-54","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27-cngrzz-18","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.49-cngrzz-40","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.43-cngrzz-34","obj_delete_at_hint.0000000014","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32:8","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-36:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.45:10","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:3","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.42-cngrzz-33","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28-cngrzz-19","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.16-cngrzz-7","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32:8","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29-cngrzz-20","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","data.full-sync.index.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.2","obj_delete_at_hint.0000000121","data_log.1","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28:5","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:5","obj_delete_at_hint.0000000125","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28:8","obj_delete_at_hint.0000000082","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-46:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.3:8","meta.full-sync.index.1","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.40-cngrzz-31","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21-cngrzz-12","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:7","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.23-cngrzz-14","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7772.9-cngrzz-58","datalog.sync-status.shard.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.1","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:1","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","meta.log.9ca21b56-c204-42f4-9360-46419e3ebcc4.0","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:1","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:3","datalog.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:1","datalog.sync-status.shard.e66a1e9b-6861-42ff-8eed-7287194f3fe0.3","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.46-cngrzz-37","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:1","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:3","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-48:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.5:8","mdlog.sync-status.shard.2","obj_delete_at_hint.0000000045","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:3","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-36:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.45:10","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-47:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.4:8","obj_delete_at_hint.0000000002","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.24-cngrzz-15","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35-cngrzz-26","obj_delete_at_hint.0000000116","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-46:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.3:8","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7240.4-cngrzz-47","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:3","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:5","obj_delete_at_hint.0000000066","data.full-sync.index.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.0","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:1","obj_delete_at_hint.0000000073","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.3-cngrzz-3","obj_delete_at_hint.0000000038","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29:8","meta.full-sync.index.3","obj_delete_at_hint.0000000118","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-15:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.24:1","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29:8","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-14:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.23:1","mdlog.sync-status.shard.3","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.38-cngrzz-29","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7772.7-cngrzz-56","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:8","mdlog.sync-status.shard.0","datalog.sync-status.shard.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.3","obj_delete_at_hint.0000000027","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:1","mdlog.sync-status","obj_delete_at_hint.0000000019","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.22-cngrzz-13","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30:8","obj_delete_at_hint.0000000039","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33:8","obj_delete_at_hint.0000000100","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:1","obj_delete_at_hint.0000000093","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7240.1-cngrzz-44","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20-cngrzz-11","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:8","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:7","data_log.0","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:5","obj_delete_at_hint.0000000012","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.18-cngrzz-9","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:3","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32-cngrzz-23","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7772.2-cngrzz-51","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-48:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.5:8","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7539.1-cngrzz-49","data.full-sync.index.e66a1e9b-6861-42ff-8eed-7287194f3fe0.0","obj_delete_at_hint.0000000072","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25-cngrzz-16","obj_delete_at_hint.0000000062","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-14:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.23:1","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","data.full-sync.index.e66a1e9b-6861-42ff-8eed-7287194f3fe0.1","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28:5","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-37:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.46:10","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-47:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.4:8","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:3","obj_delete_at_hint.0000000060","data.full-sync.index.e66a1e9b-6861-42ff-8eed-7287194f3fe0.2","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30-cngrzz-21","data.full-sync.index.e66a1e9b-6861-42ff-8eed-7287194f3fe0.3","obj_delete_at_hint.0000000051","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:8","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:7","obj_delete_at_hint.0000000106","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.37-cngrzz-28","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:7","obj_delete_at_hint.0000000017","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-15:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.24:1","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34-cngrzz-25","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:5","obj_delete_at_hint.0000000023","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-49:5fd641b5-9375-4754-8474-97ebd79eafb6.7539.1:5","2020-07-13-06-7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.1-cngrzz-42","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:5","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","meta.full-sync.index.2","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.44-cngrzz-35","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:3","datalog.sync-status.shard.e66a1e9b-6861-42ff-8eed-7287194f3fe0.1","data_log.2","obj_delete_at_hint.0000000067","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34:8","obj_delete_at_hint.0000000086","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.1-cngrzz-1","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:8","datalog.sync-status.shard.e66a1e9b-6861-42ff-8eed-7287194f3fe0.0","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34:8","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.41-cngrzz-32","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-13:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.22:1","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:5","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.36-cngrzz-27","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:1","obj_delete_at_hint.0000000124","data.full-sync.index.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.3","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:7","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:1","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","obj_delete_at_hint.0000000075","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:8","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-38:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.47:10"] 2020-07-13T06:32:48.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:48.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:48.959 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:32:48.997 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:48.998 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:49.240 INFO:teuthology.orchestra.run.smithi014.stdout:["bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:8","obj_delete_at_hint.0000000078","meta.history","2020-07-13-06-7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.2-cngrzz-43","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33:8","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30:8","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:3","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:3","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.17-cngrzz-8","obj_delete_at_hint.0000000028","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:7","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28:8","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:3","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31:8","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.47-cngrzz-38","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26-cngrzz-17","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-47:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.4:8","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-37:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.46:10","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33-cngrzz-24","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:5","obj_delete_at_hint.0000000024","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.6-cngrzz-6","datalog.sync-status.shard.5fd641b5-9375-4754-8474-97ebd79eafb6.2","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:1","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7240.2-cngrzz-45","data_log.3","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-37:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.46:10","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.50-cngrzz-41","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31-cngrzz-22","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:1","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-13:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.22:1","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:7","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.45-cngrzz-36","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7240.5-cngrzz-48","datalog.sync-status.shard.e66a1e9b-6861-42ff-8eed-7287194f3fe0.2","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-48:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.5:8","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-38:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.47:10","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7240.3-cngrzz-46","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.48-cngrzz-39","obj_delete_at_hint.0000000110","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-14:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.23:1","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19-cngrzz-10","datalog.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0","obj_delete_at_hint.0000000096","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35:8","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.2-cngrzz-2","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.4-cngrzz-4","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:3","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7772.8-cngrzz-57","obj_delete_at_hint.0000000022","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.5-cngrzz-5","bilog.trim","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:3","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:1","obj_delete_at_hint.0000000016","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7772.5-cngrzz-54","obj_delete_at_hint.0000000041","datalog.sync-status.shard.5fd641b5-9375-4754-8474-97ebd79eafb6.0","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27-cngrzz-18","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.49-cngrzz-40","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.43-cngrzz-34","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:3","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34:8","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28-cngrzz-19","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.16-cngrzz-7","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32:8","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29-cngrzz-20","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28:5","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:5","obj_delete_at_hint.0000000125","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28:8","obj_delete_at_hint.0000000082","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-46:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.3:8","meta.full-sync.index.1","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.40-cngrzz-31","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21-cngrzz-12","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:7","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.23-cngrzz-14","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:3","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:1","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","meta.log.9ca21b56-c204-42f4-9360-46419e3ebcc4.0","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:1","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:1","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:3","datalog.sync-status.shard.e66a1e9b-6861-42ff-8eed-7287194f3fe0.3","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.46-cngrzz-37","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32:8","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-48:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.5:8","mdlog.sync-status.shard.2","data.full-sync.index.5fd641b5-9375-4754-8474-97ebd79eafb6.1","obj_delete_at_hint.0000000045","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-36:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.45:10","obj_delete_at_hint.0000000002","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.24-cngrzz-15","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35-cngrzz-26","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30:8","obj_delete_at_hint.0000000116","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:1","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35:8","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:5","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7240.4-cngrzz-47","obj_delete_at_hint.0000000066","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-15:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.24:1","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","obj_delete_at_hint.0000000073","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.3-cngrzz-3","obj_delete_at_hint.0000000038","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28:5","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29:8","data.full-sync.index.5fd641b5-9375-4754-8474-97ebd79eafb6.2","meta.full-sync.index.3","obj_delete_at_hint.0000000118","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-15:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.24:1","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-14:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.23:1","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:7","obj_delete_at_hint.0000000098","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:8","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:1","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:1","mdlog.sync-status","obj_delete_at_hint.0000000019","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.22-cngrzz-13","obj_delete_at_hint.0000000039","data.full-sync.index.5fd641b5-9375-4754-8474-97ebd79eafb6.3","obj_delete_at_hint.0000000100","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:8","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:1","obj_delete_at_hint.0000000093","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7240.1-cngrzz-44","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20-cngrzz-11","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:8","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33:8","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","data_log.0","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-46:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.3:8","obj_delete_at_hint.0000000012","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.18-cngrzz-9","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:5","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:3","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32-cngrzz-23","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7772.2-cngrzz-51","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7539.1-cngrzz-49","data.full-sync.index.5fd641b5-9375-4754-8474-97ebd79eafb6.0","data.full-sync.index.e66a1e9b-6861-42ff-8eed-7287194f3fe0.0","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-36:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.45:10","obj_delete_at_hint.0000000072","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:3","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25-cngrzz-16","obj_delete_at_hint.0000000062","datalog.sync-status.shard.5fd641b5-9375-4754-8474-97ebd79eafb6.1","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","data.full-sync.index.e66a1e9b-6861-42ff-8eed-7287194f3fe0.1","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-47:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.4:8","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:3","obj_delete_at_hint.0000000060","data.full-sync.index.e66a1e9b-6861-42ff-8eed-7287194f3fe0.2","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30-cngrzz-21","data.full-sync.index.e66a1e9b-6861-42ff-8eed-7287194f3fe0.3","datalog.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6","obj_delete_at_hint.0000000051","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:8","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:7","obj_delete_at_hint.0000000106","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.37-cngrzz-28","obj_delete_at_hint.0000000017","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29:8","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34-cngrzz-25","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:5","obj_delete_at_hint.0000000023","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-49:5fd641b5-9375-4754-8474-97ebd79eafb6.7539.1:5","2020-07-13-06-7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.1-cngrzz-42","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:7","obj_delete_at_hint.0000000107","datalog.sync-status.shard.5fd641b5-9375-4754-8474-97ebd79eafb6.3","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-38:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.47:10","obj_delete_at_hint.0000000046","meta.full-sync.index.2","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:8","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:3","datalog.sync-status.shard.e66a1e9b-6861-42ff-8eed-7287194f3fe0.1","data_log.2","obj_delete_at_hint.0000000067","obj_delete_at_hint.0000000086","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.1-cngrzz-1","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31:8","datalog.sync-status.shard.e66a1e9b-6861-42ff-8eed-7287194f3fe0.0","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34:8","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-49:5fd641b5-9375-4754-8474-97ebd79eafb6.7539.1:5","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-13:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.22:1","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","bucket.sync-status.e66a1e9b-6861-42ff-8eed-7287194f3fe0:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:5","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:1","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:1","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030"] 2020-07-13T06:32:49.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:49.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:49.272 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:32:49.309 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:49.310 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:49.573 INFO:teuthology.orchestra.run.smithi017.stdout:["bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:8","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:1","obj_delete_at_hint.0000000078","meta.history","2020-07-13-06-7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.2-cngrzz-43","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:3","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:3","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.17-cngrzz-8","obj_delete_at_hint.0000000028","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:7","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28:8","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:3","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-13:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.22:1","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:3","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.47-cngrzz-38","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26-cngrzz-17","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-47:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.4:8","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33-cngrzz-24","meta.log.c0ad3871-076a-4736-ba2c-6cf14ab7b93e.0","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:5","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31:8","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:8","obj_delete_at_hint.0000000024","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:1","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.6-cngrzz-6","datalog.sync-status.shard.5fd641b5-9375-4754-8474-97ebd79eafb6.2","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:1","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7240.2-cngrzz-45","data_log.3","datalog.sync-status.shard.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.0","datalog.sync-status.shard.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.2","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-49:5fd641b5-9375-4754-8474-97ebd79eafb6.7539.1:5","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-37:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.46:10","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.50-cngrzz-41","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31-cngrzz-22","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-13:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.22:1","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.45-cngrzz-36","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7240.5-cngrzz-48","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-48:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.5:8","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7772.3-cngrzz-52","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7240.3-cngrzz-46","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.48-cngrzz-39","obj_delete_at_hint.0000000110","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-14:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.23:1","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19-cngrzz-10","obj_delete_at_hint.0000000096","data.full-sync.index.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.1","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7772.6-cngrzz-55","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.2-cngrzz-2","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.4-cngrzz-4","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:3","obj_delete_at_hint.0000000006","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28:8","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35:8","obj_delete_at_hint.0000000022","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.5-cngrzz-5","bilog.trim","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:1","obj_delete_at_hint.0000000016","obj_delete_at_hint.0000000041","datalog.sync-status.shard.5fd641b5-9375-4754-8474-97ebd79eafb6.0","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27-cngrzz-18","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.49-cngrzz-40","obj_delete_at_hint.0000000014","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32:8","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-36:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.45:10","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34:8","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28-cngrzz-19","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.16-cngrzz-7","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29-cngrzz-20","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","data.full-sync.index.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.2","obj_delete_at_hint.0000000121","data_log.1","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21-cngrzz-12","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.23-cngrzz-14","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7772.9-cngrzz-58","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:3","datalog.sync-status.shard.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.1","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","meta.log.587bd98b-69d2-49ba-9dbd-2be72177aade.1","meta.log.9ca21b56-c204-42f4-9360-46419e3ebcc4.0","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:3","datalog.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:3","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.46-cngrzz-37","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:1","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:3","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32:8","mdlog.sync-status.shard.2","data.full-sync.index.5fd641b5-9375-4754-8474-97ebd79eafb6.1","obj_delete_at_hint.0000000045","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:3","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-47:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.4:8","obj_delete_at_hint.0000000002","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.24-cngrzz-15","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35-cngrzz-26","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30:8","obj_delete_at_hint.0000000116","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:1","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35:8","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-46:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.3:8","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:5","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7240.4-cngrzz-47","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:3","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:5","obj_delete_at_hint.0000000066","data.full-sync.index.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.0","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-15:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.24:1","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:1","obj_delete_at_hint.0000000073","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.3-cngrzz-3","obj_delete_at_hint.0000000038","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28:5","data.full-sync.index.5fd641b5-9375-4754-8474-97ebd79eafb6.2","meta.full-sync.index.3","obj_delete_at_hint.0000000118","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29:8","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.38-cngrzz-29","obj_delete_at_hint.0000000119","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:7","obj_delete_at_hint.0000000098","mdlog.sync-status.shard.0","datalog.sync-status.shard.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.3","obj_delete_at_hint.0000000027","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:1","mdlog.sync-status","obj_delete_at_hint.0000000019","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.22-cngrzz-13","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30:8","obj_delete_at_hint.0000000039","meta.log.587bd98b-69d2-49ba-9dbd-2be72177aade.0","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33:8","data.full-sync.index.5fd641b5-9375-4754-8474-97ebd79eafb6.3","obj_delete_at_hint.0000000100","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:8","obj_delete_at_hint.0000000093","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7240.1-cngrzz-44","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20-cngrzz-11","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33:8","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:7","data_log.0","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:5","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-46:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.3:8","obj_delete_at_hint.0000000012","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.18-cngrzz-9","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:5","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32-cngrzz-23","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-48:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.5:8","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.7539.1-cngrzz-49","data.full-sync.index.5fd641b5-9375-4754-8474-97ebd79eafb6.0","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-36:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.45:10","obj_delete_at_hint.0000000072","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:3","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25-cngrzz-16","obj_delete_at_hint.0000000062","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-14:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.23:1","datalog.sync-status.shard.5fd641b5-9375-4754-8474-97ebd79eafb6.1","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28:5","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-37:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.46:10","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30-cngrzz-21","datalog.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:7","obj_delete_at_hint.0000000017","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-15:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.24:1","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29:8","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34-cngrzz-25","obj_delete_at_hint.0000000023","2020-07-13-06-7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.6383.1-cngrzz-42","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:7","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:5","obj_delete_at_hint.0000000107","datalog.sync-status.shard.5fd641b5-9375-4754-8474-97ebd79eafb6.3","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-38:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.47:10","obj_delete_at_hint.0000000046","meta.full-sync.index.2","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.44-cngrzz-35","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:8","data_log.2","obj_delete_at_hint.0000000067","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34:8","obj_delete_at_hint.0000000086","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.1-cngrzz-1","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:8","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31:8","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-49:5fd641b5-9375-4754-8474-97ebd79eafb6.7539.1:5","2020-07-13-06-5fd641b5-9375-4754-8474-97ebd79eafb6.4473.41-cngrzz-32","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:1","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:1","obj_delete_at_hint.0000000124","data.full-sync.index.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e.3","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:7","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:1","obj_delete_at_hint.0000000061","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:1","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","obj_delete_at_hint.0000000075","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:8","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","bucket.sync-status.7e9e4b65-ecb3-4e45-8f5d-a2651ce5636e:cngrzz-38:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.47:10"] 2020-07-13T06:32:49.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:49.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:49.596 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:32:49.636 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:49.636 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:49.894 INFO:teuthology.orchestra.run.smithi014.stdout:["bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:8","obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:3","obj_delete_at_hint.0000000028","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:7","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28:8","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:3","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-47:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.4:8","meta.log.c0ad3871-076a-4736-ba2c-6cf14ab7b93e.0","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:5","obj_delete_at_hint.0000000024","datalog.sync-status.shard.5fd641b5-9375-4754-8474-97ebd79eafb6.2","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:1","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-37:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.46:10","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-13:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.22:1","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-48:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.5:8","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-14:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.23:1","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:3","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bilog.trim","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:1","obj_delete_at_hint.0000000016","obj_delete_at_hint.0000000041","datalog.sync-status.shard.5fd641b5-9375-4754-8474-97ebd79eafb6.0","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-25:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.34:8","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:3","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","meta.log.587bd98b-69d2-49ba-9dbd-2be72177aade.1","meta.log.9ca21b56-c204-42f4-9360-46419e3ebcc4.0","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:3","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-23:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.32:8","mdlog.sync-status.shard.2","data.full-sync.index.5fd641b5-9375-4754-8474-97ebd79eafb6.1","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-21:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.30:8","obj_delete_at_hint.0000000116","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-18:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.27:1","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-26:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.35:8","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:5","obj_delete_at_hint.0000000066","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-15:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.24:1","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-19:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.28:5","data.full-sync.index.5fd641b5-9375-4754-8474-97ebd79eafb6.2","meta.full-sync.index.3","obj_delete_at_hint.0000000118","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:7","obj_delete_at_hint.0000000098","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-17:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.26:1","mdlog.sync-status","obj_delete_at_hint.0000000019","obj_delete_at_hint.0000000039","meta.log.587bd98b-69d2-49ba-9dbd-2be72177aade.0","data.full-sync.index.5fd641b5-9375-4754-8474-97ebd79eafb6.3","obj_delete_at_hint.0000000100","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:8","obj_delete_at_hint.0000000093","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-24:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.33:8","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-46:5fd641b5-9375-4754-8474-97ebd79eafb6.7240.3:8","obj_delete_at_hint.0000000012","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-16:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.25:5","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","data.full-sync.index.5fd641b5-9375-4754-8474-97ebd79eafb6.0","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-36:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.45:10","obj_delete_at_hint.0000000072","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-11:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.20:3","obj_delete_at_hint.0000000062","datalog.sync-status.shard.5fd641b5-9375-4754-8474-97ebd79eafb6.1","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","datalog.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-20:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.29:8","obj_delete_at_hint.0000000023","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:7","obj_delete_at_hint.0000000107","datalog.sync-status.shard.5fd641b5-9375-4754-8474-97ebd79eafb6.3","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-38:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.47:10","obj_delete_at_hint.0000000046","meta.full-sync.index.2","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:8","obj_delete_at_hint.0000000067","obj_delete_at_hint.0000000086","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-22:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.31:8","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-49:5fd641b5-9375-4754-8474-97ebd79eafb6.7539.1:5","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-12:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.21:1","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","bucket.sync-status.5fd641b5-9375-4754-8474-97ebd79eafb6:cngrzz-10:5fd641b5-9375-4754-8474-97ebd79eafb6.4473.19:1","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030"] 2020-07-13T06:32:50.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:50.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:50.019 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:32:50.058 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:50.058 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:50.523 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621968.545427_200.3","last_update":"2020-07-13T06:32:48.545427Z"},{"marker":"1_1594621968.545427_534.3","last_update":"2020-07-13T06:32:48.545427Z"},{"marker":"1_1594621968.545427_333.2","last_update":"2020-07-13T06:32:48.545427Z"},{"marker":"1_1594621968.545427_553.3","last_update":"2020-07-13T06:32:48.545427Z"}] 2020-07-13T06:32:50.524 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-13T06:32:50.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:50.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:50.526 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:32:50.569 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:50.569 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:50.812 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15669168566042326328},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:32:50.813 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:32:55.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:55.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:32:55.816 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:32:55.858 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:32:55.858 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:32:56.101 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15669168566042326328},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:32:56.102 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:33:01.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:01.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:01.105 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:33:01.147 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:33:01.147 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:33:01.399 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15669168566042326328},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:33:01.399 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:33:06.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:06.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:06.403 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:33:06.447 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:33:06.448 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:33:06.689 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15669168566042326328},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:33:06.690 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:33:11.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:11.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:11.694 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:33:11.739 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:33:11.740 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:33:11.980 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15669168566042326328},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:33:11.981 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:33:13.238 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.232+0000 7f6a5e7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.238 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.232+0000 7f6a537f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.239 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.232+0000 7f6a6dffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.239 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.232+0000 7f6a53ff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.240 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.232+0000 7f6a67fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.242 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a64ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.242 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a50ff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.242 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a677fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.245 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a66ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.245 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a54ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.246 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a557fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.246 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a5ffff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.246 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a527f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.246 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a51ff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.246 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a56ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.247 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a517f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.247 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a6d7fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.247 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a5f7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.247 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a567fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.247 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a547f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.248 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a52ff5700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.248 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a667fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.248 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a65ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.248 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a5cff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.248 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a5effd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.249 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a5e7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.249 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a537f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.249 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a6cff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.249 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a657fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.250 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a5dffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.250 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a6dffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.250 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a53ff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:13.250 INFO:tasks.rgw.c2.client.1.smithi014.stdout:2020-07-13T06:33:13.236+0000 7f6a67fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:16.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:16.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:16.984 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:33:17.030 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:33:17.031 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:33:17.277 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15669168566042326328},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:33:17.278 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:33:22.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:22.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:22.280 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:33:22.324 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:33:22.324 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:33:22.583 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15669168566042326328},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:33:22.584 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:33:27.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:27.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:27.587 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:33:27.635 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:33:27.635 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:33:27.870 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15669168566042326328},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:33:27.871 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:33:32.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:32.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:32.874 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:33:32.917 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:33:32.918 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:33:33.174 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15669168566042326328},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:33:33.175 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:33:38.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:38.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:38.178 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:33:38.224 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:33:38.225 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:33:38.475 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15669168566042326328},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:33:38.476 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:33:43.007 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.000+0000 7f5713fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.007 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.000+0000 7f56fcff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.008 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.000+0000 7f5708ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.008 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.000+0000 7f57117fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.009 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.000+0000 7f57027fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.009 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.000+0000 7f56fd7f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.009 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.000+0000 7f5702ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.013 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.000+0000 7f5710ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.013 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.000+0000 7f5701ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.014 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.000+0000 7f5719ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.014 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.000+0000 7f5703fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.014 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.004+0000 7f570b7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.017 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.004+0000 7f57007f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.017 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.004+0000 7f57127fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.018 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.004+0000 7f57017fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.018 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.004+0000 7f56fe7f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.019 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.004+0000 7f56ff7f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.019 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.004+0000 7f5711ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.019 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.004+0000 7f56ffff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.020 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.004+0000 7f5708ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.020 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.004+0000 7f57117fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.020 INFO:tasks.rgw.c2.client.0.smithi014.stdout:2020-07-13T06:33:43.004+0000 7f57027fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:33:43.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:43.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:43.479 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:33:43.525 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:33:43.526 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:33:43.774 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15669168566042326328},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621968.545427_200.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:32:48.545427Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621968.545427_534.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:32:48.545427Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621968.545427_333.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:32:48.545427Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621968.545427_553.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:32:48.545427Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:33:43.775 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-13T06:33:43.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:43.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:43.776 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:33:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:33:43.816 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:33:44.078 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621968.545427_200.3","last_update":"2020-07-13T06:32:48.545427Z"},{"marker":"1_1594621968.545427_534.3","last_update":"2020-07-13T06:32:48.545427Z"},{"marker":"1_1594621968.545427_333.2","last_update":"2020-07-13T06:32:48.545427Z"},{"marker":"1_1594621968.545427_553.3","last_update":"2020-07-13T06:32:48.545427Z"}] 2020-07-13T06:33:44.079 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-13T06:33:44.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:44.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:44.081 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:33:44.166 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:33:44.166 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:33:44.410 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2389703693262437130},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:33:44.411 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:33:49.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:49.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:49.415 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:33:49.461 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:33:49.462 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:33:49.692 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2389703693262437130},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:33:49.693 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:33:54.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:54.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:54.697 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:33:54.741 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:33:54.741 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:33:54.988 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2389703693262437130},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:33:54.989 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:33:59.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:59.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:33:59.993 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:00.036 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:00.037 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:00.260 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2389703693262437130},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:34:00.261 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:34:05.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:05.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:05.264 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:05.306 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:05.307 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:05.542 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2389703693262437130},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:34:05.543 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:34:10.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:10.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:10.546 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:10.585 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:10.585 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:10.827 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2389703693262437130},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:34:10.828 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:34:15.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:15.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:15.831 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:15.873 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:15.874 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:16.134 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2389703693262437130},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:34:16.134 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:34:21.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:21.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:21.137 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:21.427 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2389703693262437130},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:34:21.428 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:34:26.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:26.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:26.432 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:26.477 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:26.477 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:26.722 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2389703693262437130},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:34:26.723 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:34:31.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:31.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:31.728 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:31.771 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:31.772 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:32.014 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2389703693262437130},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:34:32.015 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:34:37.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:37.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:37.019 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:37.063 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:37.064 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:37.325 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2389703693262437130},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:34:37.326 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:34:42.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:42.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:42.329 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:42.372 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:42.373 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:42.617 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2389703693262437130},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621829.154035_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:30:29.154035Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621486.213353_328.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:24:46.213353Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621137.230195_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:18:57.230195Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621636.558054_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:27:16.558054Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:34:42.618 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1594621968.545427_200.3 target=1_1594621829.154035_138.1, shard=1 master=1_1594621968.545427_534.3 target=1_1594621486.213353_328.1, shard=2 master=1_1594621968.545427_333.2 target=1_1594621137.230195_96.1, shard=3 master=1_1594621968.545427_553.3 target=1_1594621636.558054_398.1 2020-07-13T06:34:43.030 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2020-07-13T06:34:43.021+0000 7facb97ea700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:34:43.031 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2020-07-13T06:34:43.021+0000 7face27fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:34:43.031 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2020-07-13T06:34:43.021+0000 7facb7fe7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:34:43.034 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2020-07-13T06:34:43.025+0000 7facbcff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:34:43.035 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2020-07-13T06:34:43.025+0000 7facc17fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:34:43.035 INFO:tasks.rgw.c1.client.1.smithi017.stdout:2020-07-13T06:34:43.025+0000 7facb9feb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-13T06:34:47.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:47.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:47.621 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:47.674 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:47.674 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:47.917 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2389703693262437130},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621968.545427_200.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:32:48.545427Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621968.545427_534.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:32:48.545427Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621968.545427_333.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:32:48.545427Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621968.545427_553.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:32:48.545427Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:34:47.918 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-13T06:34:47.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:47.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:47.922 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:48.004 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:48.004 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:48.260 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594621968.545427_200.3","last_update":"2020-07-13T06:32:48.545427Z"},{"marker":"1_1594621968.545427_534.3","last_update":"2020-07-13T06:32:48.545427Z"},{"marker":"1_1594621968.545427_333.2","last_update":"2020-07-13T06:32:48.545427Z"},{"marker":"1_1594621968.545427_553.3","last_update":"2020-07-13T06:32:48.545427Z"}] 2020-07-13T06:34:48.261 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-13T06:34:48.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:48.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:48.263 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:48.306 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:48.306 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:48.680 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12943611757838484443},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594621968.545427_200.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:32:48.545427Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594621968.545427_534.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:32:48.545427Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594621968.545427_333.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:32:48.545427Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594621968.545427_553.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:32:48.545427Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:34:48.681 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-13T06:34:48.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:48.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:48.683 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:48.761 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:48.762 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:49.006 INFO:teuthology.orchestra.run.smithi014.stdout:[{"marker":"1_1594622083.546247_174.1","last_update":"2020-07-13T06:34:43.546247Z"},{"marker":"1_1594622083.549409_512.1","last_update":"2020-07-13T06:34:43.549409Z"},{"marker":"1_1594622083.539843_315.1","last_update":"2020-07-13T06:34:43.539843Z"},{"marker":"1_1594622083.543144_438.1","last_update":"2020-07-13T06:34:43.543144Z"}] 2020-07-13T06:34:49.007 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-13T06:34:49.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:49.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:49.009 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:49.051 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:49.052 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:49.299 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6157611387314156630},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594622083.546247_174.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:34:43.546247Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594622083.549409_512.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:34:43.549409Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594622083.539843_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:34:43.539843Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594622083.543144_438.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:34:43.543144Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:34:49.300 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-13T06:34:49.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:49.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:49.302 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:49.342 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:49.343 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:49.597 INFO:teuthology.orchestra.run.smithi014.stdout:[{"marker":"1_1594622083.546247_174.1","last_update":"2020-07-13T06:34:43.546247Z"},{"marker":"1_1594622083.549409_512.1","last_update":"2020-07-13T06:34:43.549409Z"},{"marker":"1_1594622083.539843_315.1","last_update":"2020-07-13T06:34:43.539843Z"},{"marker":"1_1594622083.543144_438.1","last_update":"2020-07-13T06:34:43.543144Z"}] 2020-07-13T06:34:49.598 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-13T06:34:49.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:49.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:49.600 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:49.647 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:49.647 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:49.901 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3188359685164663541},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594622083.546247_174.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:34:43.546247Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594622083.549409_512.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:34:43.549409Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594622083.539843_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:34:43.539843Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594622083.543144_438.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:34:43.543144Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:34:49.902 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-13T06:34:49.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:49.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:49.904 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:49.982 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:49.983 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:50.246 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594622083.541222_166.1","last_update":"2020-07-13T06:34:43.541222Z"},{"marker":"1_1594622083.544369_426.1","last_update":"2020-07-13T06:34:43.544369Z"},{"marker":"1_1594622083.535260_210.1","last_update":"2020-07-13T06:34:43.535260Z"},{"marker":"1_1594622083.538056_526.1","last_update":"2020-07-13T06:34:43.538056Z"}] 2020-07-13T06:34:50.246 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-13T06:34:50.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:50.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:50.248 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:50.329 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:50.330 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:50.605 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3540715737640754362},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594622083.541222_166.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:34:43.541222Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594622083.544369_426.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:34:43.544369Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594622083.535260_210.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:34:43.535260Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594622083.538056_526.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:34:43.538056Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:34:50.606 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-13T06:34:50.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:50.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:50.608 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:50.686 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:50.686 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:50.934 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594622083.541222_166.1","last_update":"2020-07-13T06:34:43.541222Z"},{"marker":"1_1594622083.544369_426.1","last_update":"2020-07-13T06:34:43.544369Z"},{"marker":"1_1594622083.535260_210.1","last_update":"2020-07-13T06:34:43.535260Z"},{"marker":"1_1594622083.538056_526.1","last_update":"2020-07-13T06:34:43.538056Z"}] 2020-07-13T06:34:50.934 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-13T06:34:50.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:50.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:50.936 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:50.982 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:50.982 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:51.239 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11134568813707895163},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594622083.541222_166.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:34:43.541222Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594622083.544369_426.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:34:43.544369Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594622083.535260_210.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:34:43.535260Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594622083.538056_526.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:34:43.538056Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:34:51.239 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-13T06:34:51.240 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... ok 2020-07-13T06:34:51.241 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-59 2020-07-13T06:34:51.295 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-60 2020-07-13T06:34:51.376 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-61 2020-07-13T06:34:52.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:52.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:52.090 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:52.136 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:52.136 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:52.414 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.51.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.56.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.31.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.34.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:34:52.415 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-59 2020-07-13T06:34:52.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:52.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:52.418 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:52.472 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:52.473 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:52.690 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5","timestamp":"2020-07-13T06:34:51.714201Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-07-13T06:34:51.718349Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.31.5","timestamp":"2020-07-13T06:34:51.674198Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.34.5","timestamp":"2020-07-13T06:34:51.714157Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:34:52.691 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-59 2020-07-13T06:34:52.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:52.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:52.693 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:52.735 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:52.736 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:52.982 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.51.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.56.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.31.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.34.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:34:52.982 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-59 2020-07-13T06:34:52.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:52.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:52.990 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:53.065 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:53.066 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:53.332 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5","timestamp":"2020-07-13T06:34:51.668690Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-07-13T06:34:51.713331Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.31.5","timestamp":"2020-07-13T06:34:51.713284Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.34.5","timestamp":"2020-07-13T06:34:51.709248Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:34:53.333 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-59 2020-07-13T06:34:53.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:53.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:53.336 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:53.414 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:53.415 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:53.657 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.51.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.56.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.31.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.34.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:34:53.658 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-59 2020-07-13T06:34:53.659 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:53.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:53.661 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:53.704 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:53.705 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:53.975 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.51.5","timestamp":"2020-07-13T06:34:51.565431Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-07-13T06:34:51.513817Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.31.5","timestamp":"2020-07-13T06:34:51.456676Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.34.5","timestamp":"2020-07-13T06:34:51.615232Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:34:53.976 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-59 2020-07-13T06:34:53.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:53.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:53.979 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:54.059 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:54.060 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:54.314 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000001.51.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.55.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.44.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.23.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:34:54.314 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-59 2020-07-13T06:34:54.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:54.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:54.317 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:54.366 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:54.367 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:54.639 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5","timestamp":"2020-07-13T06:34:51.984490Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-07-13T06:34:52.021260Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5","timestamp":"2020-07-13T06:34:51.983600Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.23.5","timestamp":"2020-07-13T06:34:51.984345Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:34:54.640 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-59 2020-07-13T06:34:54.641 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:54.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:54.642 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:54.683 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:54.683 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:54.938 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000001.51.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.55.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.44.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.23.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:34:54.939 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-59 2020-07-13T06:34:54.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:54.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:54.941 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:54.985 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:55.017 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:55.247 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5","timestamp":"2020-07-13T06:34:51.989949Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-07-13T06:34:52.061214Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5","timestamp":"2020-07-13T06:34:51.989790Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.23.5","timestamp":"2020-07-13T06:34:52.033218Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:34:55.248 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-59 2020-07-13T06:34:55.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:55.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:55.249 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:55.336 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:55.337 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:55.572 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.48.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.54.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.29.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.24.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:34:55.572 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-59 2020-07-13T06:34:55.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:55.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:55.574 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:55.656 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:55.656 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:55.928 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5","timestamp":"2020-07-13T06:34:51.922340Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5","timestamp":"2020-07-13T06:34:51.965220Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.29.5","timestamp":"2020-07-13T06:34:51.921414Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.24.5","timestamp":"2020-07-13T06:34:51.965200Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:34:55.929 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-59 2020-07-13T06:34:55.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:55.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:55.932 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:56.018 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:56.019 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:56.256 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.48.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.54.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.29.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.24.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:34:56.257 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-59 2020-07-13T06:34:56.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:56.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:34:56.260 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:34:56.304 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:34:56.305 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:34:56.581 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5","timestamp":"2020-07-13T06:34:51.920297Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5","timestamp":"2020-07-13T06:34:51.962222Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.29.5","timestamp":"2020-07-13T06:34:51.919058Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.24.5","timestamp":"2020-07-13T06:34:51.962188Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:34:56.582 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-59 2020-07-13T06:34:56.582 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-59 zones={test-zone2, test-zone1} 2020-07-13T06:34:56.793 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:34:56.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:34:56.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:34:56.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:34:56.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:34:56.805 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:34:56.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:34:56.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:34:56.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:34:56.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:34:56.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:34:56.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:34:57.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:34:57.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:34:57.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:34:57.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:34:57.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:34:57.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:34:57.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:34:57.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:34:57.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:34:57.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:34:57.977 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-59 zones={test-zone2, test-zone1} 2020-07-13T06:34:57.978 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-59 zones={test-zone3, test-zone1} 2020-07-13T06:34:58.661 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:34:58.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:34:58.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:34:58.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:34:58.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:34:58.680 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:34:58.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:34:58.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:34:58.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:34:58.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:34:58.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:34:58.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:34:58.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:34:59.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:34:59.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:34:59.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:34:59.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:34:59.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:34:59.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:34:59.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:34:59.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:34:59.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:00.025 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-59 zones={test-zone3, test-zone1} 2020-07-13T06:35:00.026 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-59 zones={test-zone3, test-zone2} 2020-07-13T06:35:00.856 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:35:00.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:00.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:00.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:00.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:00.872 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:35:00.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:00.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:00.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:00.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:00.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:00.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:01.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:01.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:01.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:01.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:01.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:01.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:01.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:01.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:01.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:01.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:02.094 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-59 zones={test-zone3, test-zone2} 2020-07-13T06:35:02.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:02.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:02.107 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:02.152 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:02.152 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:02.400 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.45.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.55.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.174.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.55.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:35:02.401 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-60 2020-07-13T06:35:02.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:02.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:02.404 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:02.449 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:02.449 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:02.701 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-07-13T06:34:52.317042Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-07-13T06:34:52.102103Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.174.5","timestamp":"2020-07-13T06:34:52.102129Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-07-13T06:34:52.358189Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:35:02.702 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-60 2020-07-13T06:35:02.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:02.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:02.705 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:02.748 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:02.748 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:03.006 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.45.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.55.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.174.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.55.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:35:03.024 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-60 2020-07-13T06:35:03.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:03.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:03.026 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:03.100 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:03.101 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:03.374 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-07-13T06:34:52.306684Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-07-13T06:34:52.105284Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.174.5","timestamp":"2020-07-13T06:34:52.105252Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-07-13T06:34:52.349181Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:35:03.375 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-60 2020-07-13T06:35:03.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:03.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:03.378 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:03.417 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:03.417 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:03.674 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.45.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.55.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.174.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.55.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:35:03.675 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-60 2020-07-13T06:35:03.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:03.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:03.677 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:03.723 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:03.724 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:03.994 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-07-13T06:34:51.773773Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-07-13T06:34:51.718358Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.174.5","timestamp":"2020-07-13T06:34:51.666001Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-07-13T06:34:51.819387Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:35:03.995 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-60 2020-07-13T06:35:03.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:03.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:03.998 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:04.081 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:04.082 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:04.331 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000001.56.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.48.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.167.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.73.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:35:04.332 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-60 2020-07-13T06:35:04.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:04.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:04.335 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:04.381 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:04.382 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:04.659 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-07-13T06:34:52.021285Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5","timestamp":"2020-07-13T06:34:51.821269Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.167.5","timestamp":"2020-07-13T06:34:51.829240Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5","timestamp":"2020-07-13T06:34:52.021294Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:35:04.660 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-60 2020-07-13T06:35:04.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:04.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:04.662 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:04.709 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:04.710 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:04.951 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000001.56.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.48.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.167.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.73.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:35:04.952 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-60 2020-07-13T06:35:04.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:04.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:04.955 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:05.000 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:05.000 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:05.267 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-07-13T06:34:52.017171Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5","timestamp":"2020-07-13T06:34:51.821327Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.167.5","timestamp":"2020-07-13T06:34:51.821298Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5","timestamp":"2020-07-13T06:34:52.025201Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:35:05.268 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-60 2020-07-13T06:35:05.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:05.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:05.270 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:05.351 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:05.352 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:05.606 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.50.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.43.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.169.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.60.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:35:05.607 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-60 2020-07-13T06:35:05.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:05.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:05.609 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:05.691 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:05.692 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:05.974 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5","timestamp":"2020-07-13T06:34:52.361149Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-07-13T06:34:52.205144Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.169.5","timestamp":"2020-07-13T06:34:52.161209Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5","timestamp":"2020-07-13T06:34:52.365171Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:35:05.975 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-60 2020-07-13T06:35:05.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:05.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:05.977 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:06.018 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:06.019 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:06.276 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.50.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.43.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.169.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.60.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:35:06.276 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-60 2020-07-13T06:35:06.277 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:06.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:06.278 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:06.324 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:06.325 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:06.582 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5","timestamp":"2020-07-13T06:34:52.366120Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-07-13T06:34:52.206031Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.169.5","timestamp":"2020-07-13T06:34:52.119209Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5","timestamp":"2020-07-13T06:34:52.366087Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:35:06.583 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-60 2020-07-13T06:35:06.584 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-60 zones={test-zone2, test-zone1} 2020-07-13T06:35:07.402 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:35:07.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:07.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:07.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:07.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:07.414 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:35:07.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:07.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:07.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:07.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:07.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:07.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:07.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:07.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:07.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:07.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:08.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:08.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:08.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:08.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:08.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:08.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:08.647 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-60 zones={test-zone2, test-zone1} 2020-07-13T06:35:08.647 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-60 zones={test-zone3, test-zone1} 2020-07-13T06:35:09.132 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:35:09.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:09.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:09.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:09.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:09.144 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:35:09.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:09.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:09.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:09.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:09.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:09.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:09.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:09.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:09.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:09.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:09.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:10.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:10.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:10.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:10.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:10.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:10.587 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-60 zones={test-zone3, test-zone1} 2020-07-13T06:35:10.588 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-60 zones={test-zone3, test-zone2} 2020-07-13T06:35:10.813 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:35:10.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:10.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:10.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:10.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:10.827 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:35:10.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:10.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:10.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:10.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:10.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:10.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:11.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:11.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:11.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:11.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:11.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:11.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:11.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:11.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:11.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:11.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:12.074 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-60 zones={test-zone3, test-zone2} 2020-07-13T06:35:12.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:12.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:12.079 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:12.122 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:12.123 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:12.370 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.57.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.41.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.41.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.70.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:35:12.371 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-61 2020-07-13T06:35:12.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:12.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:12.373 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:12.419 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:12.420 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:12.687 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-07-13T06:34:52.358156Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5","timestamp":"2020-07-13T06:34:52.501976Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5","timestamp":"2020-07-13T06:34:52.316201Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.70.5","timestamp":"2020-07-13T06:34:52.546060Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:35:12.688 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-61 2020-07-13T06:35:12.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:12.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:12.690 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:12.736 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:12.736 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:13.000 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.57.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.41.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.41.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.70.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:35:13.001 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-61 2020-07-13T06:35:13.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:13.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:13.003 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:13.088 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:13.088 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:13.352 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-07-13T06:34:52.345151Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5","timestamp":"2020-07-13T06:34:52.501731Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5","timestamp":"2020-07-13T06:34:52.306708Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.70.5","timestamp":"2020-07-13T06:34:52.549235Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:35:13.353 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-61 2020-07-13T06:35:13.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:13.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:13.356 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:13.397 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:13.398 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:13.647 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.57.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.41.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.41.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.70.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:35:13.647 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-61 2020-07-13T06:35:13.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:13.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:13.649 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:13.697 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:13.710 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:13.968 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-07-13T06:34:51.982465Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.41.5","timestamp":"2020-07-13T06:34:51.923797Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.41.5","timestamp":"2020-07-13T06:34:51.872043Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.70.5","timestamp":"2020-07-13T06:34:52.043603Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:35:13.969 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-61 2020-07-13T06:35:13.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:13.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:13.973 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:14.015 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:14.016 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:14.276 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000001.75.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.35.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.57.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:35:14.277 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-61 2020-07-13T06:35:14.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:14.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:14.279 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:14.324 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:14.324 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:14.580 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5","timestamp":"2020-07-13T06:34:52.421319Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5","timestamp":"2020-07-13T06:34:52.421326Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5","timestamp":"2020-07-13T06:34:52.265279Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-07-13T06:34:52.421296Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:35:14.580 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-61 2020-07-13T06:35:14.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:14.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:14.582 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:14.625 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:14.625 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:14.886 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000001.75.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.35.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.57.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:35:14.887 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-61 2020-07-13T06:35:14.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:14.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:14.889 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:14.934 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:14.934 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:15.215 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5","timestamp":"2020-07-13T06:34:52.421223Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5","timestamp":"2020-07-13T06:34:52.461196Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5","timestamp":"2020-07-13T06:34:52.229479Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-07-13T06:34:52.425513Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:35:15.215 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-61 2020-07-13T06:35:15.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:15.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:15.217 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:15.298 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:15.299 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:15.556 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.58.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.44.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.30.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.56.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:35:15.557 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-61 2020-07-13T06:35:15.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:15.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:15.559 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:15.646 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:15.646 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:15.927 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5","timestamp":"2020-07-13T06:34:52.122911Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5","timestamp":"2020-07-13T06:34:52.209179Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.5","timestamp":"2020-07-13T06:34:51.961180Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-07-13T06:34:52.209200Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:35:15.927 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-61 2020-07-13T06:35:15.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:15.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:15.929 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:16.014 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:16.015 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:16.265 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.58.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.44.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.30.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.56.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:35:16.266 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-61 2020-07-13T06:35:16.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:16.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:16.268 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:16.322 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:16.323 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:16.583 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5","timestamp":"2020-07-13T06:34:52.162044Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5","timestamp":"2020-07-13T06:34:52.202138Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.5","timestamp":"2020-07-13T06:34:51.962231Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-07-13T06:34:52.202158Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:35:16.584 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-61 2020-07-13T06:35:16.584 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-61 zones={test-zone2, test-zone1} 2020-07-13T06:35:18.169 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:35:18.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:18.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:18.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:18.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:18.182 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:35:18.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:18.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:18.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:18.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:18.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:18.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:18.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:18.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:18.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:18.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:18.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:18.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:18.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:19.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:19.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:19.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:19.392 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-61 zones={test-zone2, test-zone1} 2020-07-13T06:35:19.393 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-61 zones={test-zone3, test-zone1} 2020-07-13T06:35:19.810 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:35:19.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:19.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:19.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:19.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:19.821 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:35:19.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:19.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:19.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:19.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:19.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:19.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:20.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:20.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:20.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:20.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:20.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:20.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:20.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:20.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:20.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:20.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:20.997 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-61 zones={test-zone3, test-zone1} 2020-07-13T06:35:20.998 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-61 zones={test-zone3, test-zone2} 2020-07-13T06:35:21.651 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:35:21.664 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:21.664 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:21.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:21.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:21.666 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:35:21.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:21.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:21.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:21.681 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:21.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:21.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:21.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:22.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:22.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:22.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:22.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:22.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:22.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:22.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:22.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:22.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:22.866 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-61 zones={test-zone3, test-zone2} 2020-07-13T06:35:22.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'cngrzz-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:35:22.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'cngrzz-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:35:22.868 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket cngrzz-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:35:22.912 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:22.913 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:23.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'cngrzz-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:35:23.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'cngrzz-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:35:23.243 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket cngrzz-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:35:23.326 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:23.326 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:23.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'cngrzz-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:35:23.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'cngrzz-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:35:23.703 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket cngrzz-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:35:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:23.780 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:24.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'cngrzz-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:35:24.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'cngrzz-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:35:24.100 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket cngrzz-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:35:24.177 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:24.178 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:24.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'cngrzz-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:35:24.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'cngrzz-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:35:24.512 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket cngrzz-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:35:24.592 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:24.592 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:24.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'cngrzz-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:35:24.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'cngrzz-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:35:24.920 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket cngrzz-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:35:25.003 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:25.003 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:27.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:27.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:27.077 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:27.138 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:27.139 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:27.392 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000008.63.3"},{"key":1,"val":"00000000004.48.1"},{"key":2,"val":"00000000006.74.5"},{"key":3,"val":"00000000004.38.1"},{"key":4,"val":"00000000006.41.5"},{"key":5,"val":"00000000004.56.1"},{"key":6,"val":"00000000004.64.1"},{"key":7,"val":"00000000006.40.5"},{"key":8,"val":"00000000004.64.1"},{"key":9,"val":"00000000004.56.1"},{"key":10,"val":"00000000004.77.1"}]} 2020-07-13T06:35:27.393 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-59 2020-07-13T06:35:27.394 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:27.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:27.396 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:27.442 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:27.442 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:27.710 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000008.63.3","timestamp":"2020-07-13T06:35:25.749888Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.74.5","timestamp":"2020-07-13T06:35:25.942026Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.38.1","timestamp":"2020-07-13T06:35:25.934018Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.41.5","timestamp":"2020-07-13T06:35:25.746140Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-07-13T06:35:25.934038Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.40.5","timestamp":"2020-07-13T06:35:25.941998Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-07-13T06:35:25.701894Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1","timestamp":"2020-07-13T06:35:25.894591Z"}}}] 2020-07-13T06:35:27.710 WARNING:rgw_multi.tests:bucket cngrzz-59 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.48.1 target=, shard=5 master=00000000004.56.1 target=, shard=9 master=00000000004.56.1 target= 2020-07-13T06:35:32.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:32.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:32.714 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:32.757 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:32.758 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:33.019 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000008.63.3","timestamp":"2020-07-13T06:35:25.749888Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.74.5","timestamp":"2020-07-13T06:35:25.942026Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.38.1","timestamp":"2020-07-13T06:35:25.934018Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.41.5","timestamp":"2020-07-13T06:35:25.746140Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-07-13T06:35:25.934038Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.40.5","timestamp":"2020-07-13T06:35:25.941998Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-07-13T06:35:25.701894Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1","timestamp":"2020-07-13T06:35:25.894591Z"}}}] 2020-07-13T06:35:33.020 WARNING:rgw_multi.tests:bucket cngrzz-59 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.48.1 target=, shard=5 master=00000000004.56.1 target=, shard=9 master=00000000004.56.1 target= 2020-07-13T06:35:38.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:38.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:38.024 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:38.076 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:38.076 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:38.342 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000008.63.3","timestamp":"2020-07-13T06:35:25.749888Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.74.5","timestamp":"2020-07-13T06:35:25.942026Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.38.1","timestamp":"2020-07-13T06:35:25.934018Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.41.5","timestamp":"2020-07-13T06:35:25.746140Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-07-13T06:35:25.934038Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.40.5","timestamp":"2020-07-13T06:35:25.941998Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-07-13T06:35:25.701894Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1","timestamp":"2020-07-13T06:35:25.894591Z"}}}] 2020-07-13T06:35:38.342 WARNING:rgw_multi.tests:bucket cngrzz-59 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.48.1 target=, shard=5 master=00000000004.56.1 target=, shard=9 master=00000000004.56.1 target= 2020-07-13T06:35:43.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:43.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:43.346 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:43.388 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:43.388 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:43.631 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000008.63.3","timestamp":"2020-07-13T06:35:25.749888Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1","timestamp":"2020-07-13T06:35:43.046152Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.74.5","timestamp":"2020-07-13T06:35:25.942026Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.38.1","timestamp":"2020-07-13T06:35:25.934018Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.41.5","timestamp":"2020-07-13T06:35:25.746140Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1","timestamp":"2020-07-13T06:35:43.004482Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-07-13T06:35:25.934038Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.40.5","timestamp":"2020-07-13T06:35:25.941998Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-07-13T06:35:25.701894Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1","timestamp":"2020-07-13T06:35:43.046166Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1","timestamp":"2020-07-13T06:35:25.894591Z"}}}] 2020-07-13T06:35:43.632 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-59 2020-07-13T06:35:43.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:43.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:43.634 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:43.680 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:43.680 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:43.931 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000008.63.3"},{"key":1,"val":"00000000004.48.1"},{"key":2,"val":"00000000006.74.5"},{"key":3,"val":"00000000004.38.1"},{"key":4,"val":"00000000006.41.5"},{"key":5,"val":"00000000004.56.1"},{"key":6,"val":"00000000004.64.1"},{"key":7,"val":"00000000006.40.5"},{"key":8,"val":"00000000004.64.1"},{"key":9,"val":"00000000004.56.1"},{"key":10,"val":"00000000004.77.1"}]} 2020-07-13T06:35:43.932 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-59 2020-07-13T06:35:43.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:43.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:43.934 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:44.019 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:44.020 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:44.279 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000008.63.3","timestamp":"2020-07-13T06:35:25.749888Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1","timestamp":"2020-07-13T06:35:43.048917Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.74.5","timestamp":"2020-07-13T06:35:25.973344Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.38.1","timestamp":"2020-07-13T06:35:25.889279Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.41.5","timestamp":"2020-07-13T06:35:25.741259Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1","timestamp":"2020-07-13T06:35:43.048948Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-07-13T06:35:25.888854Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.40.5","timestamp":"2020-07-13T06:35:25.973352Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-07-13T06:35:25.688184Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1","timestamp":"2020-07-13T06:35:43.049312Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1","timestamp":"2020-07-13T06:35:25.930157Z"}}}] 2020-07-13T06:35:44.280 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-59 2020-07-13T06:35:44.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:44.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:44.282 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:44.322 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:44.323 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:44.576 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000008.63.3"},{"key":1,"val":"00000000004.48.1"},{"key":2,"val":"00000000006.74.5"},{"key":3,"val":"00000000004.38.1"},{"key":4,"val":"00000000006.41.5"},{"key":5,"val":"00000000004.56.1"},{"key":6,"val":"00000000004.64.1"},{"key":7,"val":"00000000006.40.5"},{"key":8,"val":"00000000004.64.1"},{"key":9,"val":"00000000004.56.1"},{"key":10,"val":"00000000004.77.1"}]} 2020-07-13T06:35:44.577 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-59 2020-07-13T06:35:44.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:44.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:44.579 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:44.623 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:44.624 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:44.897 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.63.3","timestamp":"2020-07-13T06:35:25.749888Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1","timestamp":"2020-07-13T06:35:23.644451Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.74.5","timestamp":"2020-07-13T06:35:25.696078Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.38.1","timestamp":"2020-07-13T06:35:23.644608Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.41.5","timestamp":"2020-07-13T06:35:25.647922Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1","timestamp":"2020-07-13T06:35:23.646235Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-07-13T06:35:23.646279Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.40.5","timestamp":"2020-07-13T06:35:25.804233Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-07-13T06:35:23.648108Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1","timestamp":"2020-07-13T06:35:23.644716Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1","timestamp":"2020-07-13T06:35:23.644928Z"}}}] 2020-07-13T06:35:44.898 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-59 2020-07-13T06:35:44.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:44.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:44.900 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:44.942 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:44.943 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:45.218 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000004.59.5"},{"key":1,"val":"00000000002.72.1"},{"key":2,"val":"00000000004.69.5"},{"key":3,"val":"00000000002.47.1"},{"key":4,"val":"00000000004.54.5"},{"key":5,"val":"00000000002.59.1"},{"key":6,"val":"00000000002.61.1"},{"key":7,"val":"00000000004.29.5"},{"key":8,"val":"00000000002.59.1"},{"key":9,"val":"00000000002.52.1"},{"key":10,"val":"00000000002.86.1"}]} 2020-07-13T06:35:45.219 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-59 2020-07-13T06:35:45.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:45.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:45.222 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:45.266 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:45.266 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:45.532 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.59.5","timestamp":"2020-07-13T06:35:43.249212Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.72.1","timestamp":"2020-07-13T06:35:43.145508Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.69.5","timestamp":"2020-07-13T06:35:43.145500Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.47.1","timestamp":"2020-07-13T06:35:43.107641Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.54.5","timestamp":"2020-07-13T06:35:43.249234Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.59.1","timestamp":"2020-07-13T06:35:43.109405Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1","timestamp":"2020-07-13T06:35:43.141580Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.29.5","timestamp":"2020-07-13T06:35:43.109422Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.59.1","timestamp":"2020-07-13T06:35:43.265303Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.52.1","timestamp":"2020-07-13T06:35:43.153261Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.86.1","timestamp":"2020-07-13T06:35:43.107668Z"}}}] 2020-07-13T06:35:45.533 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-59 2020-07-13T06:35:45.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:45.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:45.535 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:45.577 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:45.577 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:45.834 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000004.59.5"},{"key":1,"val":"00000000002.72.1"},{"key":2,"val":"00000000004.69.5"},{"key":3,"val":"00000000002.47.1"},{"key":4,"val":"00000000004.54.5"},{"key":5,"val":"00000000002.59.1"},{"key":6,"val":"00000000002.61.1"},{"key":7,"val":"00000000004.29.5"},{"key":8,"val":"00000000002.59.1"},{"key":9,"val":"00000000002.52.1"},{"key":10,"val":"00000000002.86.1"}]} 2020-07-13T06:35:45.835 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-59 2020-07-13T06:35:45.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:45.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:45.838 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:45.885 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:46.145 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.59.5","timestamp":"2020-07-13T06:35:43.217230Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.72.1","timestamp":"2020-07-13T06:35:43.125995Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.69.5","timestamp":"2020-07-13T06:35:43.257302Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.47.1","timestamp":"2020-07-13T06:35:43.165176Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.54.5","timestamp":"2020-07-13T06:35:43.213949Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.59.1","timestamp":"2020-07-13T06:35:43.169562Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1","timestamp":"2020-07-13T06:35:43.253399Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.29.5","timestamp":"2020-07-13T06:35:43.141303Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.59.1","timestamp":"2020-07-13T06:35:43.217259Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.52.1","timestamp":"2020-07-13T06:35:43.137496Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.86.1","timestamp":"2020-07-13T06:35:43.257269Z"}}}] 2020-07-13T06:35:46.146 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-59 2020-07-13T06:35:46.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:46.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:46.149 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:46.226 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:46.227 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:46.472 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000004.56.5"},{"key":1,"val":"00000000002.46.1"},{"key":2,"val":"00000000004.70.5"},{"key":3,"val":"00000000002.32.1"},{"key":4,"val":"00000000004.39.5"},{"key":5,"val":"00000000002.49.1"},{"key":6,"val":"00000000002.60.1"},{"key":7,"val":"00000000004.30.5"},{"key":8,"val":"00000000002.55.1"},{"key":9,"val":"00000000002.40.1"},{"key":10,"val":"00000000002.83.1"}]} 2020-07-13T06:35:46.473 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-59 2020-07-13T06:35:46.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:46.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:46.476 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:46.590 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:46.591 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:46.816 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.56.5","timestamp":"2020-07-13T06:35:43.257292Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1","timestamp":"2020-07-13T06:35:43.072273Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.70.5","timestamp":"2020-07-13T06:35:43.137206Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.32.1","timestamp":"2020-07-13T06:35:43.161245Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.39.5","timestamp":"2020-07-13T06:35:43.257255Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1","timestamp":"2020-07-13T06:35:43.068764Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.60.1","timestamp":"2020-07-13T06:35:43.122220Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.30.5","timestamp":"2020-07-13T06:35:43.125846Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1","timestamp":"2020-07-13T06:35:43.257285Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1","timestamp":"2020-07-13T06:35:43.122187Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.83.1","timestamp":"2020-07-13T06:35:43.137183Z"}}}] 2020-07-13T06:35:46.817 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-59 2020-07-13T06:35:46.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:46.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:46.819 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:46.862 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:46.863 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:47.103 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000004.56.5"},{"key":1,"val":"00000000002.46.1"},{"key":2,"val":"00000000004.70.5"},{"key":3,"val":"00000000002.32.1"},{"key":4,"val":"00000000004.39.5"},{"key":5,"val":"00000000002.49.1"},{"key":6,"val":"00000000002.60.1"},{"key":7,"val":"00000000004.30.5"},{"key":8,"val":"00000000002.55.1"},{"key":9,"val":"00000000002.40.1"},{"key":10,"val":"00000000002.83.1"}]} 2020-07-13T06:35:47.107 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-59 2020-07-13T06:35:47.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:47.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:47.109 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:47.157 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:47.157 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:47.466 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.56.5","timestamp":"2020-07-13T06:35:42.980666Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1","timestamp":"2020-07-13T06:35:43.021992Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.70.5","timestamp":"2020-07-13T06:35:42.980719Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.32.1","timestamp":"2020-07-13T06:35:42.976691Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.39.5","timestamp":"2020-07-13T06:35:42.979969Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1","timestamp":"2020-07-13T06:35:42.976734Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.60.1","timestamp":"2020-07-13T06:35:42.980755Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.30.5","timestamp":"2020-07-13T06:35:42.978509Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1","timestamp":"2020-07-13T06:35:42.980737Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1","timestamp":"2020-07-13T06:35:42.976368Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.83.1","timestamp":"2020-07-13T06:35:42.979940Z"}}}] 2020-07-13T06:35:47.467 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-59 2020-07-13T06:35:47.468 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-59 zones={test-zone2, test-zone1} 2020-07-13T06:35:48.104 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:35:48.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:48.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:48.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:48.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:48.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:35:48.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:35:48.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:35:48.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:35:48.118 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:35:48.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:48.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:48.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:48.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:48.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:35:48.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:35:48.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:35:48.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:35:48.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:48.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:48.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:48.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:48.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:48.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:48.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:48.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:48.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:49.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:49.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:49.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:49.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:35:49.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:35:49.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:35:49.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:35:49.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:35:49.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:35:49.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:35:50.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:35:50.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:35:50.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:35:50.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:35:50.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:35:50.586 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-59 zones={test-zone2, test-zone1} 2020-07-13T06:35:50.587 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-59 zones={test-zone3, test-zone1} 2020-07-13T06:35:51.712 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:35:51.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:51.722 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:51.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:51.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:51.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:35:51.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:35:51.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:35:51.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:35:51.729 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:35:51.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:51.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:51.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:51.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:51.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:35:51.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:35:51.751 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:35:51.751 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:35:51.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:52.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:52.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:52.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:52.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:52.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:52.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:52.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:52.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:52.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:53.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:53.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:53.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:35:53.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:35:53.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:35:53.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:35:53.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:35:53.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:35:53.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:35:53.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:35:53.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:35:54.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:35:54.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:35:54.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:35:54.393 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-59 zones={test-zone3, test-zone1} 2020-07-13T06:35:54.394 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-59 zones={test-zone3, test-zone2} 2020-07-13T06:35:55.016 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:35:55.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:55.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:55.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:55.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:55.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:35:55.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:35:55.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:35:55.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:35:55.034 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:35:55.043 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:35:55.044 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:35:55.045 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:35:55.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:35:55.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:35:55.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:35:55.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:35:55.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:35:55.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:55.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:55.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:35:55.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:55.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:55.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:35:55.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:55.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:55.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:35:55.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:55.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:56.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:35:56.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:35:56.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:35:56.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:35:56.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:35:56.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:35:56.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:35:56.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:35:56.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:35:57.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:35:57.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:35:57.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:35:57.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:35:57.443 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-59 zones={test-zone3, test-zone2} 2020-07-13T06:35:57.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:57.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:57.447 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:57.491 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:57.492 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:57.745 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.51.5"},{"key":1,"val":"00000000004.64.1"},{"key":2,"val":"00000000006.61.5"},{"key":3,"val":"00000000004.68.1"},{"key":4,"val":"00000000006.180.5"},{"key":5,"val":"00000000004.49.1"},{"key":6,"val":"00000000004.71.1"},{"key":7,"val":"00000000006.71.5"},{"key":8,"val":"00000000004.84.1"},{"key":9,"val":"00000000004.85.1"},{"key":10,"val":"00000000004.72.1"}]} 2020-07-13T06:35:57.746 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-60 2020-07-13T06:35:57.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:57.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:57.749 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:57.795 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:57.796 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:58.066 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.51.5","timestamp":"2020-07-13T06:35:26.468168Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-07-13T06:35:43.042167Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.61.5","timestamp":"2020-07-13T06:35:26.421302Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1","timestamp":"2020-07-13T06:35:25.898648Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.180.5","timestamp":"2020-07-13T06:35:26.372223Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1","timestamp":"2020-07-13T06:35:43.046116Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1","timestamp":"2020-07-13T06:35:25.942019Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.71.5","timestamp":"2020-07-13T06:35:26.521438Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.84.1","timestamp":"2020-07-13T06:35:25.697741Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.85.1","timestamp":"2020-07-13T06:35:43.003219Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1","timestamp":"2020-07-13T06:35:25.938128Z"}}}] 2020-07-13T06:35:58.067 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-60 2020-07-13T06:35:58.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:58.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:58.070 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:58.110 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:58.111 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:58.365 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.51.5"},{"key":1,"val":"00000000004.64.1"},{"key":2,"val":"00000000006.61.5"},{"key":3,"val":"00000000004.68.1"},{"key":4,"val":"00000000006.180.5"},{"key":5,"val":"00000000004.49.1"},{"key":6,"val":"00000000004.71.1"},{"key":7,"val":"00000000006.71.5"},{"key":8,"val":"00000000004.84.1"},{"key":9,"val":"00000000004.85.1"},{"key":10,"val":"00000000004.72.1"}]} 2020-07-13T06:35:58.366 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-60 2020-07-13T06:35:58.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:58.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:58.368 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:58.449 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:58.449 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:58.722 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.51.5","timestamp":"2020-07-13T06:35:26.468168Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-07-13T06:35:43.085183Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.61.5","timestamp":"2020-07-13T06:35:26.421302Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1","timestamp":"2020-07-13T06:35:25.937162Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.180.5","timestamp":"2020-07-13T06:35:26.372223Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1","timestamp":"2020-07-13T06:35:43.047051Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1","timestamp":"2020-07-13T06:35:25.930108Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.71.5","timestamp":"2020-07-13T06:35:26.521438Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.84.1","timestamp":"2020-07-13T06:35:25.687394Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.85.1","timestamp":"2020-07-13T06:35:43.047011Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1","timestamp":"2020-07-13T06:35:25.933169Z"}}}] 2020-07-13T06:35:58.723 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-60 2020-07-13T06:35:58.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:58.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:58.725 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:58.817 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:58.818 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:59.073 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.51.5"},{"key":1,"val":"00000000004.64.1"},{"key":2,"val":"00000000006.61.5"},{"key":3,"val":"00000000004.68.1"},{"key":4,"val":"00000000006.180.5"},{"key":5,"val":"00000000004.49.1"},{"key":6,"val":"00000000004.71.1"},{"key":7,"val":"00000000006.71.5"},{"key":8,"val":"00000000004.84.1"},{"key":9,"val":"00000000004.85.1"},{"key":10,"val":"00000000004.72.1"}]} 2020-07-13T06:35:59.073 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-60 2020-07-13T06:35:59.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:59.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:59.075 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:59.121 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:59.122 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:59.394 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.51.5","timestamp":"2020-07-13T06:35:26.468168Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-07-13T06:35:24.455586Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.61.5","timestamp":"2020-07-13T06:35:26.421302Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1","timestamp":"2020-07-13T06:35:24.455574Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.180.5","timestamp":"2020-07-13T06:35:26.372223Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1","timestamp":"2020-07-13T06:35:24.458134Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1","timestamp":"2020-07-13T06:35:24.455866Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.71.5","timestamp":"2020-07-13T06:35:26.521438Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.84.1","timestamp":"2020-07-13T06:35:24.459636Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.85.1","timestamp":"2020-07-13T06:35:24.461163Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1","timestamp":"2020-07-13T06:35:24.459251Z"}}}] 2020-07-13T06:35:59.395 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-60 2020-07-13T06:35:59.396 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:59.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:59.397 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:59.480 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:59.480 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:35:59.733 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000002.60.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.52.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.171.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.85.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:35:59.738 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-60 2020-07-13T06:35:59.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:59.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:35:59.742 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:35:59.788 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:35:59.789 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:00.065 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.60.5","timestamp":"2020-07-13T06:35:26.468168Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.52.5","timestamp":"2020-07-13T06:35:26.421302Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.171.5","timestamp":"2020-07-13T06:35:26.372223Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.85.5","timestamp":"2020-07-13T06:35:26.521438Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:00.066 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-60 2020-07-13T06:36:00.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:00.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:00.068 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:00.119 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:00.120 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:00.365 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000002.60.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.52.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.171.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.85.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:00.366 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-60 2020-07-13T06:36:00.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:00.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:00.389 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:00.437 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:00.438 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:00.705 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.60.5","timestamp":"2020-07-13T06:35:26.468168Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.52.5","timestamp":"2020-07-13T06:35:26.421302Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.171.5","timestamp":"2020-07-13T06:35:26.372223Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.85.5","timestamp":"2020-07-13T06:35:26.521438Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:00.706 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-60 2020-07-13T06:36:00.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:00.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:00.709 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:00.787 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:00.788 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:01.038 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000002.54.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.47.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.173.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.70.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:01.038 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-60 2020-07-13T06:36:01.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:01.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:01.040 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:01.139 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:01.140 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:01.417 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.54.5","timestamp":"2020-07-13T06:35:26.468168Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.47.5","timestamp":"2020-07-13T06:35:26.421302Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.173.5","timestamp":"2020-07-13T06:35:26.372223Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.70.5","timestamp":"2020-07-13T06:35:26.521438Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:01.417 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-60 2020-07-13T06:36:01.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:01.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:01.419 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:01.499 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:01.500 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:01.751 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000002.54.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.47.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.173.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.70.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:01.751 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-60 2020-07-13T06:36:01.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:01.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:01.753 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:01.801 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:01.802 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:02.093 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.54.5","timestamp":"2020-07-13T06:35:26.468168Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.47.5","timestamp":"2020-07-13T06:35:26.421302Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.173.5","timestamp":"2020-07-13T06:35:26.372223Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.70.5","timestamp":"2020-07-13T06:35:26.521438Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:02.094 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-60 2020-07-13T06:36:02.094 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-60 zones={test-zone2, test-zone1} 2020-07-13T06:36:03.470 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:36:03.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:03.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:03.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:03.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:03.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:36:03.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:36:03.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:36:03.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:36:03.487 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:36:03.497 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:03.498 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:03.498 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:03.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:03.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:36:03.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:36:03.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:36:03.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:36:03.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:03.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:03.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:03.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:03.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:04.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:04.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:04.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:04.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:04.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:04.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:04.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:04.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:36:04.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:36:04.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:36:04.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:36:05.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:36:05.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:36:05.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:36:05.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:36:05.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:36:05.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:36:05.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:36:05.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:36:05.864 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-60 zones={test-zone2, test-zone1} 2020-07-13T06:36:05.864 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-60 zones={test-zone3, test-zone1} 2020-07-13T06:36:06.733 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:36:06.740 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:06.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:06.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:06.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:06.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:36:06.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:36:06.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:36:06.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:36:06.744 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:36:06.751 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:06.751 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:06.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:06.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:06.753 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:36:06.753 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:36:06.754 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:36:06.754 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:36:06.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:06.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:06.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:07.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:07.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:07.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:07.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:07.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:07.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:07.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:07.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:07.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:07.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:36:07.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:36:08.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:36:08.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:36:08.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:36:08.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:36:08.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:36:08.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:36:08.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:36:08.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:36:08.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:36:08.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:36:09.109 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-60 zones={test-zone3, test-zone1} 2020-07-13T06:36:09.112 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-60 zones={test-zone3, test-zone2} 2020-07-13T06:36:09.210 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:36:09.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:09.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:09.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:09.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:09.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:36:09.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:36:09.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:36:09.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:36:09.226 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:36:09.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:09.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:09.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:09.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:09.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:36:09.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:36:09.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:36:09.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:36:09.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:09.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:09.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:09.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:09.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:09.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:09.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:09.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:10.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:10.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:10.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:10.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:10.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:36:10.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:36:10.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:36:10.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:36:10.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:36:10.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:36:10.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:36:11.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:36:11.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:36:11.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:36:11.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:36:11.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:36:11.597 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-60 zones={test-zone3, test-zone2} 2020-07-13T06:36:11.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:11.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:11.599 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:11.641 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:11.642 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:11.887 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000007.75.5"},{"key":1,"val":"00000000004.59.1"},{"key":2,"val":"00000000007.49.5"},{"key":3,"val":"00000000004.50.1"},{"key":4,"val":"00000000007.49.5"},{"key":5,"val":"00000000004.68.1"},{"key":6,"val":"00000000004.60.1"},{"key":7,"val":"00000000007.78.5"},{"key":8,"val":"00000000004.87.1"},{"key":9,"val":"00000000004.53.1"},{"key":10,"val":"00000000004.72.1"}]} 2020-07-13T06:36:11.888 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-61 2020-07-13T06:36:11.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:11.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:11.890 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:11.936 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:11.936 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:12.200 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000007.75.5","timestamp":"2020-07-13T06:35:26.979489Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1","timestamp":"2020-07-13T06:35:43.006127Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000007.49.5","timestamp":"2020-07-13T06:35:26.929913Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1","timestamp":"2020-07-13T06:35:25.898451Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000007.49.5","timestamp":"2020-07-13T06:35:26.880232Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1","timestamp":"2020-07-13T06:35:43.005252Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1","timestamp":"2020-07-13T06:35:25.904583Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000007.78.5","timestamp":"2020-07-13T06:35:27.029126Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.87.1","timestamp":"2020-07-13T06:35:25.703901Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1","timestamp":"2020-07-13T06:35:43.005564Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1","timestamp":"2020-07-13T06:35:25.907851Z"}}}] 2020-07-13T06:36:12.201 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-61 2020-07-13T06:36:12.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:12.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:12.203 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:12.243 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:12.244 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:12.495 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000007.75.5"},{"key":1,"val":"00000000004.59.1"},{"key":2,"val":"00000000007.49.5"},{"key":3,"val":"00000000004.50.1"},{"key":4,"val":"00000000007.49.5"},{"key":5,"val":"00000000004.68.1"},{"key":6,"val":"00000000004.60.1"},{"key":7,"val":"00000000007.78.5"},{"key":8,"val":"00000000004.87.1"},{"key":9,"val":"00000000004.53.1"},{"key":10,"val":"00000000004.72.1"}]} 2020-07-13T06:36:12.496 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-61 2020-07-13T06:36:12.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:12.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:12.499 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:12.584 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:12.584 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:12.847 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000007.75.5","timestamp":"2020-07-13T06:35:26.979489Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1","timestamp":"2020-07-13T06:35:43.097494Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000007.49.5","timestamp":"2020-07-13T06:35:26.929913Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1","timestamp":"2020-07-13T06:35:25.899043Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000007.49.5","timestamp":"2020-07-13T06:35:26.880232Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1","timestamp":"2020-07-13T06:35:43.117170Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1","timestamp":"2020-07-13T06:35:25.896006Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000007.78.5","timestamp":"2020-07-13T06:35:27.029126Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.87.1","timestamp":"2020-07-13T06:35:25.688205Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1","timestamp":"2020-07-13T06:35:43.105162Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1","timestamp":"2020-07-13T06:35:25.899981Z"}}}] 2020-07-13T06:36:12.848 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-61 2020-07-13T06:36:12.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:12.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:12.850 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:12.936 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:12.936 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:13.181 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000007.75.5"},{"key":1,"val":"00000000004.59.1"},{"key":2,"val":"00000000007.49.5"},{"key":3,"val":"00000000004.50.1"},{"key":4,"val":"00000000007.49.5"},{"key":5,"val":"00000000004.68.1"},{"key":6,"val":"00000000004.60.1"},{"key":7,"val":"00000000007.78.5"},{"key":8,"val":"00000000004.87.1"},{"key":9,"val":"00000000004.53.1"},{"key":10,"val":"00000000004.72.1"}]} 2020-07-13T06:36:13.181 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-61 2020-07-13T06:36:13.182 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:13.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:13.184 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:13.232 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:13.252 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:13.494 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.75.5","timestamp":"2020-07-13T06:35:26.979489Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1","timestamp":"2020-07-13T06:35:25.255767Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.49.5","timestamp":"2020-07-13T06:35:26.929913Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1","timestamp":"2020-07-13T06:35:25.257665Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.49.5","timestamp":"2020-07-13T06:35:26.880232Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1","timestamp":"2020-07-13T06:35:25.256221Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1","timestamp":"2020-07-13T06:35:25.255212Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.78.5","timestamp":"2020-07-13T06:35:27.029126Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.87.1","timestamp":"2020-07-13T06:35:25.258513Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1","timestamp":"2020-07-13T06:35:25.257488Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1","timestamp":"2020-07-13T06:35:25.255430Z"}}}] 2020-07-13T06:36:13.495 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-61 2020-07-13T06:36:13.496 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:13.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:13.497 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:13.537 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:13.538 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:13.799 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000004.87.5"},{"key":1,"val":"00000000002.60.1"},{"key":2,"val":"00000000004.55.5"},{"key":3,"val":"00000000002.54.1"},{"key":4,"val":"00000000004.41.5"},{"key":5,"val":"00000000002.55.1"},{"key":6,"val":"00000000002.63.1"},{"key":7,"val":"00000000004.63.5"},{"key":8,"val":"00000000002.98.1"},{"key":9,"val":"00000000002.51.1"},{"key":10,"val":"00000000002.70.1"}]} 2020-07-13T06:36:13.799 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-61 2020-07-13T06:36:13.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:13.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:13.801 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:13.854 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:13.854 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:14.119 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.87.5","timestamp":"2020-07-13T06:35:43.265350Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.60.1","timestamp":"2020-07-13T06:35:43.129674Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.55.5","timestamp":"2020-07-13T06:35:26.929913Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1","timestamp":"2020-07-13T06:35:43.093472Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.41.5","timestamp":"2020-07-13T06:35:43.265338Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1","timestamp":"2020-07-13T06:35:43.095544Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1","timestamp":"2020-07-13T06:35:43.129664Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.63.5","timestamp":"2020-07-13T06:35:27.029126Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.98.1","timestamp":"2020-07-13T06:35:43.261199Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1","timestamp":"2020-07-13T06:35:43.095524Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.70.1","timestamp":"2020-07-13T06:35:43.145481Z"}}}] 2020-07-13T06:36:14.120 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-61 2020-07-13T06:36:14.121 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:14.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:14.122 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:14.167 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:14.168 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:14.422 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000004.87.5"},{"key":1,"val":"00000000002.60.1"},{"key":2,"val":"00000000004.55.5"},{"key":3,"val":"00000000002.54.1"},{"key":4,"val":"00000000004.41.5"},{"key":5,"val":"00000000002.55.1"},{"key":6,"val":"00000000002.63.1"},{"key":7,"val":"00000000004.63.5"},{"key":8,"val":"00000000002.98.1"},{"key":9,"val":"00000000002.51.1"},{"key":10,"val":"00000000002.70.1"}]} 2020-07-13T06:36:14.423 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-61 2020-07-13T06:36:14.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:14.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:14.425 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:14.471 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:14.471 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:14.752 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.87.5","timestamp":"2020-07-13T06:35:43.183082Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.60.1","timestamp":"2020-07-13T06:35:43.109241Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.55.5","timestamp":"2020-07-13T06:35:43.214021Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1","timestamp":"2020-07-13T06:35:43.123206Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.41.5","timestamp":"2020-07-13T06:35:43.225421Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1","timestamp":"2020-07-13T06:35:43.108973Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1","timestamp":"2020-07-13T06:35:43.217940Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.63.5","timestamp":"2020-07-13T06:35:27.029126Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.98.1","timestamp":"2020-07-13T06:35:43.183029Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1","timestamp":"2020-07-13T06:35:43.125977Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.70.1","timestamp":"2020-07-13T06:35:43.257293Z"}}}] 2020-07-13T06:36:14.753 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-61 2020-07-13T06:36:14.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:14.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:14.758 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:14.842 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:14.843 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:15.094 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000004.72.5"},{"key":1,"val":"00000000002.57.1"},{"key":2,"val":"00000000004.50.5"},{"key":3,"val":"00000000002.56.1"},{"key":4,"val":"00000000004.36.5"},{"key":5,"val":"00000000002.48.1"},{"key":6,"val":"00000000002.53.1"},{"key":7,"val":"00000000004.62.5"},{"key":8,"val":"00000000002.105.1"},{"key":9,"val":"00000000002.39.1"},{"key":10,"val":"00000000002.64.1"}]} 2020-07-13T06:36:15.095 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-61 2020-07-13T06:36:15.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:15.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:15.097 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:15.179 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:15.180 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:15.460 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.72.5","timestamp":"2020-07-13T06:35:43.273561Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.57.1","timestamp":"2020-07-13T06:35:43.078209Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.50.5","timestamp":"2020-07-13T06:35:43.133152Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.56.1","timestamp":"2020-07-13T06:35:43.126067Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.36.5","timestamp":"2020-07-13T06:35:43.273485Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.48.1","timestamp":"2020-07-13T06:35:43.078273Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.53.1","timestamp":"2020-07-13T06:35:43.125887Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.62.5","timestamp":"2020-07-13T06:35:43.169465Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.105.1","timestamp":"2020-07-13T06:35:43.273551Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1","timestamp":"2020-07-13T06:35:43.078234Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.64.1","timestamp":"2020-07-13T06:35:43.125868Z"}}}] 2020-07-13T06:36:15.461 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-61 2020-07-13T06:36:15.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:15.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:15.463 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:15.549 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:15.549 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:15.776 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000004.72.5"},{"key":1,"val":"00000000002.57.1"},{"key":2,"val":"00000000004.50.5"},{"key":3,"val":"00000000002.56.1"},{"key":4,"val":"00000000004.36.5"},{"key":5,"val":"00000000002.48.1"},{"key":6,"val":"00000000002.53.1"},{"key":7,"val":"00000000004.62.5"},{"key":8,"val":"00000000002.105.1"},{"key":9,"val":"00000000002.39.1"},{"key":10,"val":"00000000002.64.1"}]} 2020-07-13T06:36:15.777 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-61 2020-07-13T06:36:15.778 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:15.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:15.779 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:15.824 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:15.825 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:16.113 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.72.5","timestamp":"2020-07-13T06:35:42.980695Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.57.1","timestamp":"2020-07-13T06:35:43.014000Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.50.5","timestamp":"2020-07-13T06:35:42.978554Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.56.1","timestamp":"2020-07-13T06:35:42.976801Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.36.5","timestamp":"2020-07-13T06:35:42.979988Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.48.1","timestamp":"2020-07-13T06:35:42.976775Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.53.1","timestamp":"2020-07-13T06:35:42.979110Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.62.5","timestamp":"2020-07-13T06:35:43.017998Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.105.1","timestamp":"2020-07-13T06:35:42.979912Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1","timestamp":"2020-07-13T06:35:43.014018Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.64.1","timestamp":"2020-07-13T06:35:42.982260Z"}}}] 2020-07-13T06:36:16.114 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-61 2020-07-13T06:36:16.116 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-61 zones={test-zone2, test-zone1} 2020-07-13T06:36:17.616 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:36:17.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:17.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:17.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:17.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:17.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:36:17.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:36:17.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:36:17.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:36:17.629 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:36:17.638 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:17.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:17.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:17.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:17.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:36:17.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:36:17.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:36:17.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:36:17.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:17.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:17.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:17.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:18.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:18.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:18.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:18.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:18.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:18.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:18.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:18.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:18.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:36:18.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:36:19.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:36:19.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:36:19.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:36:19.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:36:19.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:36:19.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:36:19.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:36:19.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:36:19.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:36:19.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:36:20.118 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-61 zones={test-zone2, test-zone1} 2020-07-13T06:36:20.119 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-61 zones={test-zone3, test-zone1} 2020-07-13T06:36:20.285 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:36:20.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:20.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:20.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:20.310 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:20.310 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:36:20.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:36:20.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:36:20.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:36:20.313 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:36:20.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:20.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:20.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:20.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:20.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:36:20.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:36:20.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:36:20.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:36:20.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:20.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:20.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:20.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:20.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:20.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:20.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:21.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:21.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:21.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:21.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:21.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:21.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:36:21.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:36:21.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:36:21.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:36:21.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:36:22.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:36:22.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:36:22.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:36:22.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:36:22.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:36:22.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:36:22.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:36:22.752 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-61 zones={test-zone3, test-zone1} 2020-07-13T06:36:22.753 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-61 zones={test-zone3, test-zone2} 2020-07-13T06:36:22.818 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:36:22.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:22.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:22.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:22.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:22.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:36:22.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:36:22.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:36:22.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:36:22.840 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:36:22.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:22.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:22.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:22.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:22.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:36:22.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:36:22.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:36:22.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:36:22.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:22.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:23.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:23.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:23.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:23.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:23.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:23.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:23.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:23.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:23.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:23.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:24.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:36:24.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:36:24.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:36:24.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:36:24.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:36:24.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:36:24.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:36:24.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:36:24.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:36:24.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:36:25.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:36:25.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:36:25.294 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-61 zones={test-zone3, test-zone2} 2020-07-13T06:36:25.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:25.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:25.298 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:25.339 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:25.339 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:25.592 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594622125.267936_225.3","last_update":"2020-07-13T06:35:25.267936Z"},{"marker":"1_1594622159.678331_607.1","last_update":"2020-07-13T06:35:59.678331Z"},{"marker":"1_1594622159.678331_414.1","last_update":"2020-07-13T06:35:59.678331Z"},{"marker":"1_1594622159.678331_658.2","last_update":"2020-07-13T06:35:59.678331Z"}] 2020-07-13T06:36:25.593 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-13T06:36:25.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:25.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:25.595 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:25.644 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:25.644 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:25.898 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15669168566042326328},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594622125.267936_225.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:25.267936Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594622159.678331_607.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:59.678331Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594622159.678331_414.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:59.678331Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594622159.678331_658.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:59.678331Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:36:25.899 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-13T06:36:25.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:25.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:25.902 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:25.946 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:25.947 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:26.204 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594622125.267936_225.3","last_update":"2020-07-13T06:35:25.267936Z"},{"marker":"1_1594622159.678331_607.1","last_update":"2020-07-13T06:35:59.678331Z"},{"marker":"1_1594622159.678331_414.1","last_update":"2020-07-13T06:35:59.678331Z"},{"marker":"1_1594622159.678331_658.2","last_update":"2020-07-13T06:35:59.678331Z"}] 2020-07-13T06:36:26.205 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-13T06:36:26.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:26.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:26.207 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:26.293 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:26.293 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:26.550 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2389703693262437130},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594622125.267936_225.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:25.267936Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594622159.678331_607.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:59.678331Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594622159.678331_414.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:59.678331Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594622159.678331_658.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:59.678331Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:36:26.551 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-13T06:36:26.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:26.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:26.552 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:26.634 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:26.634 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:26.872 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594622125.267936_225.3","last_update":"2020-07-13T06:35:25.267936Z"},{"marker":"1_1594622159.678331_607.1","last_update":"2020-07-13T06:35:59.678331Z"},{"marker":"1_1594622159.678331_414.1","last_update":"2020-07-13T06:35:59.678331Z"},{"marker":"1_1594622159.678331_658.2","last_update":"2020-07-13T06:35:59.678331Z"}] 2020-07-13T06:36:26.872 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-13T06:36:26.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:26.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:26.874 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:26.922 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:26.923 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:27.196 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12943611757838484443},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594622125.267936_225.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:25.267936Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594622159.678331_607.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:59.678331Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594622159.678331_414.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:59.678331Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594622159.678331_658.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:59.678331Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:36:27.196 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-13T06:36:27.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:27.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:27.198 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:27.276 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:27.277 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:27.523 INFO:teuthology.orchestra.run.smithi014.stdout:[{"marker":"1_1594622137.784654_183.6","last_update":"2020-07-13T06:35:37.784654Z"},{"marker":"1_1594622159.804044_557.1","last_update":"2020-07-13T06:35:59.804044Z"},{"marker":"1_1594622159.804044_366.1","last_update":"2020-07-13T06:35:59.804044Z"},{"marker":"1_1594622159.804044_493.2","last_update":"2020-07-13T06:35:59.804044Z"}] 2020-07-13T06:36:27.524 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-13T06:36:27.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:27.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:27.525 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:27.573 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:27.574 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:27.822 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6157611387314156630},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594622137.784654_183.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:37.784654Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594622159.804044_557.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:59.804044Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594622159.804044_366.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:59.804044Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594622159.804044_493.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:59.804044Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:36:27.822 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-13T06:36:27.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:27.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:27.824 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:27.866 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:27.866 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:28.126 INFO:teuthology.orchestra.run.smithi014.stdout:[{"marker":"1_1594622137.784654_183.6","last_update":"2020-07-13T06:35:37.784654Z"},{"marker":"1_1594622159.804044_557.1","last_update":"2020-07-13T06:35:59.804044Z"},{"marker":"1_1594622159.804044_366.1","last_update":"2020-07-13T06:35:59.804044Z"},{"marker":"1_1594622159.804044_493.2","last_update":"2020-07-13T06:35:59.804044Z"}] 2020-07-13T06:36:28.127 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-13T06:36:28.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:28.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:28.129 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:28.171 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:28.172 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:28.429 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3188359685164663541},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594622137.784654_183.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:37.784654Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594622159.804044_557.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:59.804044Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594622159.804044_366.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:59.804044Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594622159.804044_493.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:59.804044Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:36:28.429 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-13T06:36:28.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:28.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:28.431 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:28.511 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:28.511 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:28.760 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594622137.341644_174.6","last_update":"2020-07-13T06:35:37.341644Z"},{"marker":"1_1594622137.341644_462.6","last_update":"2020-07-13T06:35:37.341644Z"},{"marker":"1_1594622137.341644_237.4","last_update":"2020-07-13T06:35:37.341644Z"},{"marker":"1_1594622137.341644_590.6","last_update":"2020-07-13T06:35:37.341644Z"}] 2020-07-13T06:36:28.761 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-13T06:36:28.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:28.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:28.764 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:28.844 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:28.845 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:29.098 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3540715737640754362},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594622137.341644_174.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:37.341644Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594622137.341644_462.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:37.341644Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594622137.341644_237.4","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:37.341644Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594622137.341644_590.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:37.341644Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:36:29.098 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-13T06:36:29.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:29.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:29.100 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:29.178 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:29.178 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:29.425 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594622137.341644_174.6","last_update":"2020-07-13T06:35:37.341644Z"},{"marker":"1_1594622137.341644_462.6","last_update":"2020-07-13T06:35:37.341644Z"},{"marker":"1_1594622137.341644_237.4","last_update":"2020-07-13T06:35:37.341644Z"},{"marker":"1_1594622137.341644_590.6","last_update":"2020-07-13T06:35:37.341644Z"}] 2020-07-13T06:36:29.426 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-13T06:36:29.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:29.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:29.429 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:29.473 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:29.473 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:29.735 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11134568813707895163},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1594622137.341644_174.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:37.341644Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1594622137.341644_462.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:37.341644Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1594622137.341644_237.4","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:37.341644Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1594622137.341644_590.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:35:37.341644Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-13T06:36:29.736 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-13T06:36:29.737 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2020-07-13T06:36:29.739 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=cngrzz-62 2020-07-13T06:36:30.130 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=cngrzz-63 2020-07-13T06:36:31.046 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=cngrzz-64 2020-07-13T06:36:34.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:34.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:34.329 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:34.372 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:34.373 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:34.575 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=9ca21b56-c204-42f4-9360-46419e3ebcc4 2020-07-13T06:36:34.606 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594622192.087229_572.1","last_update":"2020-07-13T06:36:32.087229Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:36:34.607 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:36:34.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:34.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:34.609 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:34.655 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:34.655 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:34.876 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594622192.087229_572.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:36:32.087229Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:36:34.877 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:36:34.877 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:36:34.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:34.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:34.879 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:34.916 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:34.916 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:35.169 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594622192.087229_572.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:36:32.087229Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:36:35.169 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:36:35.170 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:36:35.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:35.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:35.172 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:35.219 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:35.219 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:35.502 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594622192.087229_572.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:36:32.087229Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:36:35.503 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:36:35.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:35.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:35.505 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:35.558 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:35.558 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:35.823 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.33.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.45.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.78.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.80.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:35.823 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-62 2020-07-13T06:36:35.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:35.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:35.825 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:35.880 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:35.881 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:36.158 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.33.5","timestamp":"2020-07-13T06:36:32.558179Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-07-13T06:36:32.602081Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5","timestamp":"2020-07-13T06:36:32.362157Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.80.5","timestamp":"2020-07-13T06:36:32.558134Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:36.159 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-62 2020-07-13T06:36:36.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:36.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:36.161 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:36.210 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:36.211 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:36.493 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.33.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.45.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.78.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.80.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:36.494 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-62 2020-07-13T06:36:36.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:36.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:36.495 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:36.582 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:36.583 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:36.869 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.33.5","timestamp":"2020-07-13T06:36:32.518299Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-07-13T06:36:32.521167Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5","timestamp":"2020-07-13T06:36:32.361291Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.80.5","timestamp":"2020-07-13T06:36:32.557268Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:36.870 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-62 2020-07-13T06:36:36.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:36.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:36.873 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:36.917 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:36.918 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:37.176 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.33.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.45.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.78.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.80.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:37.177 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-62 2020-07-13T06:36:37.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:37.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:37.179 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:37.224 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:37.225 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:37.492 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.33.5","timestamp":"2020-07-13T06:36:32.371822Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-07-13T06:36:32.319428Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.78.5","timestamp":"2020-07-13T06:36:32.271473Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.80.5","timestamp":"2020-07-13T06:36:32.419208Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:37.493 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-62 2020-07-13T06:36:37.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:37.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:37.496 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:37.575 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:37.575 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:37.813 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000001.32.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.58.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.90.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.85.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:37.814 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-62 2020-07-13T06:36:37.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:37.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:37.817 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:37.864 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:37.865 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:38.150 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.32.5","timestamp":"2020-07-13T06:36:32.869288Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5","timestamp":"2020-07-13T06:36:32.905281Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.90.5","timestamp":"2020-07-13T06:36:32.665245Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-07-13T06:36:32.869327Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:38.151 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-62 2020-07-13T06:36:38.152 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:38.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:38.153 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:38.193 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:38.194 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:38.451 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000001.32.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.58.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.90.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.85.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:38.452 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-62 2020-07-13T06:36:38.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:38.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:38.454 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:38.498 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:38.499 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:38.759 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.32.5","timestamp":"2020-07-13T06:36:32.869306Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5","timestamp":"2020-07-13T06:36:32.905271Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.90.5","timestamp":"2020-07-13T06:36:32.665281Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-07-13T06:36:32.869285Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:38.760 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-62 2020-07-13T06:36:38.761 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:38.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:38.762 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:38.844 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:38.845 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:39.103 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.30.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.44.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.75.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.92.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:39.105 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-62 2020-07-13T06:36:39.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:39.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:39.108 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:39.192 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:39.192 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:39.464 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.5","timestamp":"2020-07-13T06:36:32.817279Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5","timestamp":"2020-07-13T06:36:32.865199Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5","timestamp":"2020-07-13T06:36:32.621234Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.92.5","timestamp":"2020-07-13T06:36:32.817315Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:39.465 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-62 2020-07-13T06:36:39.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:39.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:39.468 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:39.506 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:39.507 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:39.759 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.30.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.44.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.75.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.92.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:39.760 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-62 2020-07-13T06:36:39.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:39.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:39.762 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:39.808 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:39.808 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:40.080 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.5","timestamp":"2020-07-13T06:36:32.818078Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5","timestamp":"2020-07-13T06:36:32.858040Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5","timestamp":"2020-07-13T06:36:32.618232Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.92.5","timestamp":"2020-07-13T06:36:32.818100Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:40.081 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-62 2020-07-13T06:36:40.082 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-62 zones={test-zone2, test-zone1} 2020-07-13T06:36:41.906 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:36:41.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:41.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:41.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:41.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:41.922 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:36:41.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:41.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:41.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:41.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:41.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:42.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:42.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:42.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:42.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:42.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:42.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:42.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:42.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:42.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:42.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:43.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:43.136 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-62 zones={test-zone2, test-zone1} 2020-07-13T06:36:43.137 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-62 zones={test-zone3, test-zone1} 2020-07-13T06:36:44.070 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:36:44.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:44.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:44.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:44.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:44.080 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:36:44.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:44.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:44.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:44.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:44.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:44.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:44.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:44.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:44.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:44.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:44.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:44.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:44.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:44.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:45.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:45.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:45.272 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-62 zones={test-zone3, test-zone1} 2020-07-13T06:36:45.273 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-62 zones={test-zone3, test-zone2} 2020-07-13T06:36:46.019 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:36:46.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:46.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:46.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:46.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:46.030 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:36:46.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:46.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:46.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:46.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:46.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:46.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:46.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:46.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:46.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:46.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:46.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:46.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:46.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:46.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:46.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:47.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:47.245 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-62 zones={test-zone3, test-zone2} 2020-07-13T06:36:47.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:47.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:47.248 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:47.289 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:47.289 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:47.559 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.55.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.53.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.38.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.67.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:47.560 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-63 2020-07-13T06:36:47.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:47.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:47.564 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:47.615 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:47.616 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:47.876 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-07-13T06:36:33.962142Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5","timestamp":"2020-07-13T06:36:34.002072Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5","timestamp":"2020-07-13T06:36:33.762114Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5","timestamp":"2020-07-13T06:36:33.962099Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:47.876 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-63 2020-07-13T06:36:47.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:47.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:47.878 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:47.921 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:48.178 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.55.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.53.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.38.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.67.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:48.179 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-63 2020-07-13T06:36:48.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:48.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:48.181 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:48.265 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:48.266 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:48.542 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-07-13T06:36:33.965212Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5","timestamp":"2020-07-13T06:36:34.001293Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5","timestamp":"2020-07-13T06:36:33.761237Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5","timestamp":"2020-07-13T06:36:33.961250Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:48.543 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-63 2020-07-13T06:36:48.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:48.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:48.579 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:48.627 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:48.628 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:48.887 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.55.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.53.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.38.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.67.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:48.888 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-63 2020-07-13T06:36:48.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:48.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:48.890 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:48.937 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:48.937 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:49.211 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-07-13T06:36:33.479298Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.53.5","timestamp":"2020-07-13T06:36:33.419755Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.38.5","timestamp":"2020-07-13T06:36:33.359582Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.67.5","timestamp":"2020-07-13T06:36:33.531599Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:49.212 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-63 2020-07-13T06:36:49.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:49.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:49.213 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:49.297 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:49.297 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:49.560 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.77.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.45.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.114.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:49.561 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-63 2020-07-13T06:36:49.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:49.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:49.563 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:49.613 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:49.614 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:49.878 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-07-13T06:36:33.665343Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.77.5","timestamp":"2020-07-13T06:36:33.709253Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-07-13T06:36:33.465326Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.114.5","timestamp":"2020-07-13T06:36:33.665312Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:49.879 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-63 2020-07-13T06:36:49.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:49.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:49.880 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:49.921 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:49.922 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:50.157 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.77.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.45.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.114.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:50.158 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-63 2020-07-13T06:36:50.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:50.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:50.160 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:50.207 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:50.208 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:50.652 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-07-13T06:36:33.665326Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.77.5","timestamp":"2020-07-13T06:36:33.705295Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-07-13T06:36:33.465375Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.114.5","timestamp":"2020-07-13T06:36:33.665360Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:50.653 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-63 2020-07-13T06:36:50.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:50.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:50.656 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:50.737 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:50.737 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:50.988 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.65.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.61.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.47.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.73.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:50.989 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-63 2020-07-13T06:36:50.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:50.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:50.991 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:51.073 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:51.073 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:51.369 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5","timestamp":"2020-07-13T06:36:34.021249Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5","timestamp":"2020-07-13T06:36:34.061173Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5","timestamp":"2020-07-13T06:36:33.821208Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5","timestamp":"2020-07-13T06:36:34.021286Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:51.371 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-63 2020-07-13T06:36:51.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:51.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:51.373 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:51.475 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:51.476 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:51.739 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.65.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.61.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.47.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.73.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:51.740 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-63 2020-07-13T06:36:51.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:51.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:51.741 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:51.797 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:51.797 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:52.060 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5","timestamp":"2020-07-13T06:36:34.022178Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5","timestamp":"2020-07-13T06:36:34.062096Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5","timestamp":"2020-07-13T06:36:33.818154Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5","timestamp":"2020-07-13T06:36:34.022134Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:52.061 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-63 2020-07-13T06:36:52.065 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-63 zones={test-zone2, test-zone1} 2020-07-13T06:36:53.458 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:36:53.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:53.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:53.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:53.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:53.473 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:36:53.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:53.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:53.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:53.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:53.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:53.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:53.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:53.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:53.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:53.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:54.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:54.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:54.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:54.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:54.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:54.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:54.703 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-63 zones={test-zone2, test-zone1} 2020-07-13T06:36:54.704 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-63 zones={test-zone3, test-zone1} 2020-07-13T06:36:55.649 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:36:55.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:55.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:55.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:55.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:55.658 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:36:55.664 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:55.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:55.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:55.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:55.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:55.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:55.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:55.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:56.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:56.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:56.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:56.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:56.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:56.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:56.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:56.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:56.924 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-63 zones={test-zone3, test-zone1} 2020-07-13T06:36:56.924 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-63 zones={test-zone3, test-zone2} 2020-07-13T06:36:57.572 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:36:57.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:57.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:57.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:57.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:57.588 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:36:57.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:36:57.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:36:57.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:36:57.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:36:57.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:57.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:57.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:36:57.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:57.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:58.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:36:58.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:58.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:58.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:36:58.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:58.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:58.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:36:58.787 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-63 zones={test-zone3, test-zone2} 2020-07-13T06:36:58.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:58.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:58.791 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:58.836 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:58.837 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:59.091 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.83.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.57.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.84.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.40.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:59.092 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-64 2020-07-13T06:36:59.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:59.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:59.094 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:59.143 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:59.143 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:59.416 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5","timestamp":"2020-07-13T06:36:34.802214Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-07-13T06:36:34.802156Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.84.5","timestamp":"2020-07-13T06:36:34.562119Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5","timestamp":"2020-07-13T06:36:34.562088Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:36:59.417 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-64 2020-07-13T06:36:59.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:59.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:59.421 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:59.467 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:59.467 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:36:59.702 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.83.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.57.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.84.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.40.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:36:59.703 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-64 2020-07-13T06:36:59.704 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:59.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:36:59.706 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:36:59.788 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:36:59.789 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:00.060 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5","timestamp":"2020-07-13T06:36:34.801224Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-07-13T06:36:34.797186Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.84.5","timestamp":"2020-07-13T06:36:34.557176Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5","timestamp":"2020-07-13T06:36:34.557194Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:37:00.061 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-64 2020-07-13T06:37:00.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:00.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:00.064 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:00.144 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:00.145 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:00.397 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.83.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.57.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.84.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.40.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:37:00.398 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-64 2020-07-13T06:37:00.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:00.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:00.400 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:00.445 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:00.445 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:00.723 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.83.5","timestamp":"2020-07-13T06:36:34.232248Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-07-13T06:36:34.178326Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.84.5","timestamp":"2020-07-13T06:36:34.124585Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.40.5","timestamp":"2020-07-13T06:36:34.284094Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:37:00.724 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-64 2020-07-13T06:37:00.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:00.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:00.728 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:00.810 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:00.810 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:01.053 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000001.69.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.64.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.85.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.47.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:37:01.054 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-64 2020-07-13T06:37:01.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:01.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:01.056 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:01.107 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:01.108 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:01.386 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.69.5","timestamp":"2020-07-13T06:36:34.669353Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5","timestamp":"2020-07-13T06:36:34.705227Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-07-13T06:36:34.469301Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5","timestamp":"2020-07-13T06:36:34.669311Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:37:01.387 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-64 2020-07-13T06:37:01.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:01.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:01.391 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:01.443 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:01.444 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:01.678 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000001.69.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.64.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.85.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.47.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:37:01.679 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-64 2020-07-13T06:37:01.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:01.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:01.681 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:01.730 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:01.730 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:01.987 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.69.5","timestamp":"2020-07-13T06:36:34.669339Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5","timestamp":"2020-07-13T06:36:34.705226Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-07-13T06:36:34.469304Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5","timestamp":"2020-07-13T06:36:34.669311Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:37:01.988 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-64 2020-07-13T06:37:01.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:01.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:01.990 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:02.069 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:02.070 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:02.320 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.68.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.67.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.65.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.49.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:37:02.321 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-64 2020-07-13T06:37:02.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:02.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:02.323 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:02.407 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:02.468 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:02.686 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.68.5","timestamp":"2020-07-13T06:36:34.417303Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5","timestamp":"2020-07-13T06:36:34.469514Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5","timestamp":"2020-07-13T06:36:34.217207Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5","timestamp":"2020-07-13T06:36:34.417345Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:37:02.687 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-64 2020-07-13T06:37:02.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:02.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:02.691 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:02.769 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:02.769 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:03.018 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000001.68.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.67.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.65.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.49.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-13T06:37:03.019 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-64 2020-07-13T06:37:03.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:03.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:03.028 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:03.082 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:03.083 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:03.366 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.68.5","timestamp":"2020-07-13T06:36:34.418193Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5","timestamp":"2020-07-13T06:36:34.462135Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5","timestamp":"2020-07-13T06:36:34.218231Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5","timestamp":"2020-07-13T06:36:34.418145Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:37:03.367 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-64 2020-07-13T06:37:03.368 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-64 zones={test-zone2, test-zone1} 2020-07-13T06:37:04.959 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:37:04.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:37:04.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:37:04.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:37:04.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:37:04.971 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:37:04.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:37:04.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:37:04.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:37:04.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:37:05.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:37:05.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:37:05.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:37:05.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:37:05.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:37:05.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:37:05.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:37:05.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:37:05.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:37:05.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:37:05.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:37:06.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:37:06.218 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-64 zones={test-zone2, test-zone1} 2020-07-13T06:37:06.229 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-64 zones={test-zone3, test-zone1} 2020-07-13T06:37:06.512 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:37:06.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:37:06.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:37:06.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:37:06.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:37:06.523 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:37:06.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:37:06.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:37:06.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:37:06.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:37:06.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:37:06.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:37:06.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:37:06.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:37:06.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:37:07.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:37:07.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:37:07.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:37:07.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:37:07.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:37:07.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:37:07.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:37:07.724 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-64 zones={test-zone3, test-zone1} 2020-07-13T06:37:07.725 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-64 zones={test-zone3, test-zone2} 2020-07-13T06:37:08.444 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:37:08.458 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:37:08.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:37:08.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:37:08.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:37:08.461 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:37:08.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:37:08.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:37:08.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:37:08.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:37:08.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:37:08.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:37:08.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:37:08.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:37:08.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:37:09.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:37:09.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:37:09.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:37:09.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:37:09.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:37:09.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:37:09.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:37:09.894 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-64 zones={test-zone3, test-zone2} 2020-07-13T06:37:09.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'cngrzz-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:37:09.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'cngrzz-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:37:09.898 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket cngrzz-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:37:09.941 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:09.942 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:10.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'cngrzz-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:37:10.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'cngrzz-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:37:10.277 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket cngrzz-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:37:10.361 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:10.361 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:10.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'cngrzz-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:37:10.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'cngrzz-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:37:10.685 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket cngrzz-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:37:10.760 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:10.761 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:11.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:11.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:11.111 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:11.193 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:11.194 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:11.428 INFO:teuthology.orchestra.run.smithi017.stderr:No --period given, using current period=9ca21b56-c204-42f4-9360-46419e3ebcc4 2020-07-13T06:37:11.463 INFO:teuthology.orchestra.run.smithi017.stdout:[{"marker":"1_1594622231.055313_690.1","last_update":"2020-07-13T06:37:11.055313Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-13T06:37:11.464 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-13T06:37:11.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:11.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:11.467 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:11.504 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:11.504 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:11.735 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594622231.055313_690.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:37:11.055313Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:37:11.735 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-13T06:37:11.736 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-13T06:37:11.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:11.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:11.738 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:11.781 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:11.781 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:12.027 INFO:teuthology.orchestra.run.smithi017.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594622231.055313_690.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:37:11.055313Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:37:12.028 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-13T06:37:12.029 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-13T06:37:12.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:12.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:12.031 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:12.076 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:12.077 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:12.345 INFO:teuthology.orchestra.run.smithi014.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9ca21b56-c204-42f4-9360-46419e3ebcc4","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1594622231.055313_690.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-13T06:37:11.055313Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1594620933.443793_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-13T06:15:33.443793Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-13T06:37:12.346 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-13T06:37:14.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'cngrzz-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:37:14.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'cngrzz-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:37:14.780 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket cngrzz-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:37:14.824 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:14.825 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:15.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'cngrzz-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:37:15.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'cngrzz-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:37:15.172 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket cngrzz-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:37:15.252 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:15.253 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:15.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'cngrzz-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:37:15.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'cngrzz-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-13T06:37:15.593 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket cngrzz-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-13T06:37:15.675 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:15.676 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:16.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:16.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:16.025 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:16.107 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:16.108 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:16.375 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.42.1"},{"key":1,"val":"00000000004.53.1"},{"key":2,"val":"00000000006.53.1"},{"key":3,"val":"00000000004.38.1"},{"key":4,"val":"00000000006.84.1"},{"key":5,"val":"00000000004.68.1"},{"key":6,"val":"00000000004.58.1"},{"key":7,"val":"00000000006.88.1"},{"key":8,"val":"00000000004.91.1"},{"key":9,"val":"00000000004.43.1"},{"key":10,"val":"00000000004.59.1"}]} 2020-07-13T06:37:16.376 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-62 2020-07-13T06:37:16.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:16.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:16.381 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:16.428 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:16.429 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:16.686 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:37:16.687 WARNING:rgw_multi.tests:bucket cngrzz-62 zone test-zone2 behind zone test-zone1: shard=0 master=00000000006.42.1 target=, shard=1 master=00000000004.53.1 target=, shard=2 master=00000000006.53.1 target=, shard=3 master=00000000004.38.1 target=, shard=4 master=00000000006.84.1 target=, shard=5 master=00000000004.68.1 target=, shard=6 master=00000000004.58.1 target=, shard=7 master=00000000006.88.1 target=, shard=8 master=00000000004.91.1 target=, shard=9 master=00000000004.43.1 target=, shard=10 master=00000000004.59.1 target= 2020-07-13T06:37:21.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:21.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:21.709 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:21.759 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:21.760 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:22.020 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-07-13T06:37:22.021 WARNING:rgw_multi.tests:bucket cngrzz-62 zone test-zone2 behind zone test-zone1: shard=0 master=00000000006.42.1 target=, shard=1 master=00000000004.53.1 target=, shard=2 master=00000000006.53.1 target=, shard=3 master=00000000004.38.1 target=, shard=4 master=00000000006.84.1 target=, shard=5 master=00000000004.68.1 target=, shard=6 master=00000000004.58.1 target=, shard=7 master=00000000006.88.1 target=, shard=8 master=00000000004.91.1 target=, shard=9 master=00000000004.43.1 target=, shard=10 master=00000000004.59.1 target= 2020-07-13T06:37:27.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:27.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:27.025 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:27.071 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:27.072 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:27.328 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.42.1","timestamp":"2020-07-13T06:37:23.048874Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1","timestamp":"2020-07-13T06:37:23.158013Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.53.1","timestamp":"2020-07-13T06:37:23.050606Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.38.1","timestamp":"2020-07-13T06:37:23.053732Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.84.1","timestamp":"2020-07-13T06:37:23.094119Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1","timestamp":"2020-07-13T06:37:23.117411Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1","timestamp":"2020-07-13T06:37:23.050515Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.88.1","timestamp":"2020-07-13T06:37:23.047948Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.91.1","timestamp":"2020-07-13T06:37:23.048101Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1","timestamp":"2020-07-13T06:37:23.117569Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1","timestamp":"2020-07-13T06:37:23.048088Z"}}}] 2020-07-13T06:37:27.329 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-62 2020-07-13T06:37:27.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:27.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:27.332 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:27.375 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:27.375 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:27.626 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.42.1"},{"key":1,"val":"00000000004.53.1"},{"key":2,"val":"00000000006.53.1"},{"key":3,"val":"00000000004.38.1"},{"key":4,"val":"00000000006.84.1"},{"key":5,"val":"00000000004.68.1"},{"key":6,"val":"00000000004.58.1"},{"key":7,"val":"00000000006.88.1"},{"key":8,"val":"00000000004.91.1"},{"key":9,"val":"00000000004.43.1"},{"key":10,"val":"00000000004.59.1"}]} 2020-07-13T06:37:27.627 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-62 2020-07-13T06:37:27.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:27.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:27.629 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:27.719 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:27.720 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:27.998 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.42.1","timestamp":"2020-07-13T06:37:23.098368Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1","timestamp":"2020-07-13T06:37:23.085252Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.53.1","timestamp":"2020-07-13T06:37:23.210743Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.38.1","timestamp":"2020-07-13T06:37:23.098385Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.84.1","timestamp":"2020-07-13T06:37:23.095937Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1","timestamp":"2020-07-13T06:37:23.045996Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1","timestamp":"2020-07-13T06:37:23.249241Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.88.1","timestamp":"2020-07-13T06:37:23.095918Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.91.1","timestamp":"2020-07-13T06:37:23.137485Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1","timestamp":"2020-07-13T06:37:23.095852Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1","timestamp":"2020-07-13T06:37:23.211275Z"}}}] 2020-07-13T06:37:27.999 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-62 2020-07-13T06:37:28.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:28.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:28.002 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:28.045 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:28.045 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:28.307 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.42.1"},{"key":1,"val":"00000000004.53.1"},{"key":2,"val":"00000000006.53.1"},{"key":3,"val":"00000000004.38.1"},{"key":4,"val":"00000000006.84.1"},{"key":5,"val":"00000000004.68.1"},{"key":6,"val":"00000000004.58.1"},{"key":7,"val":"00000000006.88.1"},{"key":8,"val":"00000000004.91.1"},{"key":9,"val":"00000000004.43.1"},{"key":10,"val":"00000000004.59.1"}]} 2020-07-13T06:37:28.307 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-62 2020-07-13T06:37:28.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:28.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:28.310 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:28.357 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:28.358 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:28.629 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.42.1","timestamp":"2020-07-13T06:37:15.116723Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.53.1","timestamp":"2020-07-13T06:37:15.116456Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.38.1","timestamp":"2020-07-13T06:37:15.117845Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.84.1","timestamp":"2020-07-13T06:37:15.116988Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1","timestamp":"2020-07-13T06:37:15.118345Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.88.1","timestamp":"2020-07-13T06:37:15.119327Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.91.1","timestamp":"2020-07-13T06:37:15.117669Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1","timestamp":"2020-07-13T06:37:15.117732Z"}}}] 2020-07-13T06:37:28.630 WARNING:rgw_multi.tests:bucket cngrzz-62 zone test-zone4 behind zone test-zone1: shard=1 master=00000000004.53.1 target=, shard=5 master=00000000004.68.1 target=, shard=9 master=00000000004.43.1 target= 2020-07-13T06:37:33.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:33.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:33.635 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:33.674 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:33.674 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:33.947 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.42.1","timestamp":"2020-07-13T06:37:15.116723Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1","timestamp":"2020-07-13T06:37:15.116948Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.53.1","timestamp":"2020-07-13T06:37:15.116456Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.38.1","timestamp":"2020-07-13T06:37:15.117845Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.84.1","timestamp":"2020-07-13T06:37:15.116988Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1","timestamp":"2020-07-13T06:37:15.117901Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1","timestamp":"2020-07-13T06:37:15.118345Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.88.1","timestamp":"2020-07-13T06:37:15.119327Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.91.1","timestamp":"2020-07-13T06:37:15.117669Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1","timestamp":"2020-07-13T06:37:15.118147Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1","timestamp":"2020-07-13T06:37:15.117732Z"}}}] 2020-07-13T06:37:33.948 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-62 2020-07-13T06:37:33.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:33.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:33.949 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:34.047 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:34.047 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:34.286 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000004.42.5"},{"key":1,"val":"00000000002.77.1"},{"key":2,"val":"00000000004.68.5"},{"key":3,"val":"00000000002.47.1"},{"key":4,"val":"00000000004.96.5"},{"key":5,"val":"00000000002.74.1"},{"key":6,"val":"00000000002.68.1"},{"key":7,"val":"00000000004.95.5"},{"key":8,"val":"00000000002.78.1"},{"key":9,"val":"00000000002.38.1"},{"key":10,"val":"00000000002.85.1"}]} 2020-07-13T06:37:34.287 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-62 2020-07-13T06:37:34.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:34.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:34.289 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:34.333 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:34.333 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:34.606 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.42.5","timestamp":"2020-07-13T06:37:30.478949Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.77.1","timestamp":"2020-07-13T06:37:30.529315Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.68.5","timestamp":"2020-07-13T06:37:30.478985Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.47.1","timestamp":"2020-07-13T06:37:30.533841Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.96.5","timestamp":"2020-07-13T06:37:30.529295Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.74.1","timestamp":"2020-07-13T06:37:30.517807Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1","timestamp":"2020-07-13T06:37:30.517779Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.95.5","timestamp":"2020-07-13T06:37:30.533863Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.78.1","timestamp":"2020-07-13T06:37:30.478996Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.38.1","timestamp":"2020-07-13T06:37:30.478974Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1","timestamp":"2020-07-13T06:37:30.513272Z"}}}] 2020-07-13T06:37:34.607 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=cngrzz-62 2020-07-13T06:37:34.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:34.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:34.609 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:34.654 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:34.654 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:34.895 INFO:teuthology.orchestra.run.smithi014.stdout:{"markers":[{"key":0,"val":"00000000004.42.5"},{"key":1,"val":"00000000002.77.1"},{"key":2,"val":"00000000004.68.5"},{"key":3,"val":"00000000002.47.1"},{"key":4,"val":"00000000004.96.5"},{"key":5,"val":"00000000002.74.1"},{"key":6,"val":"00000000002.68.1"},{"key":7,"val":"00000000004.95.5"},{"key":8,"val":"00000000002.78.1"},{"key":9,"val":"00000000002.38.1"},{"key":10,"val":"00000000002.85.1"}]} 2020-07-13T06:37:34.895 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-62 2020-07-13T06:37:34.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:34.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:34.897 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket cngrzz-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:34.941 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:34.941 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:35.222 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.42.5","timestamp":"2020-07-13T06:37:30.516671Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.77.1","timestamp":"2020-07-13T06:37:30.516736Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.68.5","timestamp":"2020-07-13T06:37:30.516682Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.47.1","timestamp":"2020-07-13T06:37:30.573700Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.96.5","timestamp":"2020-07-13T06:37:30.573754Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.74.1","timestamp":"2020-07-13T06:37:30.518230Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1","timestamp":"2020-07-13T06:37:30.518187Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.95.5","timestamp":"2020-07-13T06:37:30.573671Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.78.1","timestamp":"2020-07-13T06:37:30.573773Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.38.1","timestamp":"2020-07-13T06:37:30.573830Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1","timestamp":"2020-07-13T06:37:30.518213Z"}}}] 2020-07-13T06:37:35.222 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=cngrzz-62 2020-07-13T06:37:35.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:35.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:35.224 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:35.306 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:35.307 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:35.572 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000004.40.5"},{"key":1,"val":"00000000002.51.1"},{"key":2,"val":"00000000004.54.5"},{"key":3,"val":"00000000002.42.1"},{"key":4,"val":"00000000004.81.5"},{"key":5,"val":"00000000002.63.1"},{"key":6,"val":"00000000002.53.1"},{"key":7,"val":"00000000004.102.5"},{"key":8,"val":"00000000002.79.1"},{"key":9,"val":"00000000002.36.1"},{"key":10,"val":"00000000002.67.1"}]} 2020-07-13T06:37:35.573 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-62 2020-07-13T06:37:35.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:35.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:35.576 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:35.658 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:35.659 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:35.942 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.40.5","timestamp":"2020-07-13T06:37:30.449190Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1","timestamp":"2020-07-13T06:37:30.678999Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.54.5","timestamp":"2020-07-13T06:37:30.623466Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.42.1","timestamp":"2020-07-13T06:37:30.627182Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.81.5","timestamp":"2020-07-13T06:37:30.406998Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1","timestamp":"2020-07-13T06:37:30.711864Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.53.1","timestamp":"2020-07-13T06:37:30.627116Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.102.5","timestamp":"2020-07-13T06:37:30.627162Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.79.1","timestamp":"2020-07-13T06:37:30.406133Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.36.1","timestamp":"2020-07-13T06:37:30.700444Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.67.1","timestamp":"2020-07-13T06:37:30.662611Z"}}}] 2020-07-13T06:37:35.943 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=cngrzz-62 2020-07-13T06:37:35.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:35.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:35.955 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:36.025 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:36.026 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:36.289 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000004.40.5"},{"key":1,"val":"00000000002.51.1"},{"key":2,"val":"00000000004.54.5"},{"key":3,"val":"00000000002.42.1"},{"key":4,"val":"00000000004.81.5"},{"key":5,"val":"00000000002.63.1"},{"key":6,"val":"00000000002.53.1"},{"key":7,"val":"00000000004.102.5"},{"key":8,"val":"00000000002.79.1"},{"key":9,"val":"00000000002.36.1"},{"key":10,"val":"00000000002.67.1"}]} 2020-07-13T06:37:36.290 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-62 2020-07-13T06:37:36.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:36.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'cngrzz-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:36.292 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket cngrzz-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:36.345 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:36.346 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:36.616 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.40.5","timestamp":"2020-07-13T06:37:30.410754Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1","timestamp":"2020-07-13T06:37:30.654623Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.54.5","timestamp":"2020-07-13T06:37:30.678030Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.42.1","timestamp":"2020-07-13T06:37:30.682027Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.81.5","timestamp":"2020-07-13T06:37:30.454020Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1","timestamp":"2020-07-13T06:37:30.678010Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.53.1","timestamp":"2020-07-13T06:37:30.654826Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.102.5","timestamp":"2020-07-13T06:37:30.654853Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.79.1","timestamp":"2020-07-13T06:37:30.413791Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.36.1","timestamp":"2020-07-13T06:37:30.654884Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.67.1","timestamp":"2020-07-13T06:37:30.686053Z"}}}] 2020-07-13T06:37:36.617 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=cngrzz-62 2020-07-13T06:37:36.618 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-62 zones={test-zone2, test-zone1} 2020-07-13T06:37:37.750 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:37:37.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:37:37.762 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:37:37.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:37:37.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:37:37.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:37:37.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:37:37.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:37:37.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:37:37.770 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:37:37.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:37:37.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:37:37.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:37:37.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:37:37.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:37:37.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:37:37.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:37:37.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:37:37.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:37:37.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:37:37.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:37:38.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:37:38.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:37:38.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:37:38.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:37:38.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:37:38.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:37:38.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:37:38.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:37:38.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:37:39.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:37:39.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:37:39.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:37:39.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:37:39.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:37:39.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:37:39.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:37:39.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:37:39.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:37:39.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:37:39.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:37:40.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:37:40.218 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-62 zones={test-zone2, test-zone1} 2020-07-13T06:37:40.219 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-62 zones={test-zone3, test-zone1} 2020-07-13T06:37:40.807 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:37:40.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:37:40.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:37:40.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:37:40.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:37:40.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:37:40.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:37:40.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:37:40.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:37:40.822 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:37:40.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:37:40.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:37:40.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:37:40.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:37:40.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:37:40.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:37:40.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:37:40.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:37:40.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:37:40.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:37:41.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:37:41.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:37:41.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:37:41.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:37:41.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:37:41.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:37:41.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:37:41.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:37:42.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:37:42.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:37:42.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:37:42.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:37:42.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:37:42.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:37:42.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:37:42.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:37:42.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:37:43.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:37:43.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:37:43.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:37:43.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:37:43.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:37:43.552 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-62 zones={test-zone3, test-zone1} 2020-07-13T06:37:43.552 INFO:tasks.rgw_multi.zone_rados:comparing bucket=cngrzz-62 zones={test-zone3, test-zone2} 2020-07-13T06:37:43.765 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-13T06:37:43.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:37:43.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:37:43.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:37:43.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:37:43.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:37:43.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:37:43.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:37:43.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:37:43.785 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-13T06:37:43.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-13T06:37:43.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-13T06:37:43.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-13T06:37:43.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-13T06:37:43.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-13T06:37:43.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-13T06:37:43.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-13T06:37:43.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-13T06:37:43.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:37:43.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:37:44.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-13T06:37:44.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:37:44.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:37:44.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-13T06:37:44.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:37:44.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:37:44.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-13T06:37:44.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:37:44.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:37:44.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-13T06:37:45.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:37:45.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:37:45.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-13T06:37:45.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:37:45.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:37:45.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-13T06:37:45.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:37:45.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:37:45.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-13T06:37:45.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:37:45.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:37:46.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-13T06:37:46.172 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=cngrzz-62 zones={test-zone3, test-zone2} 2020-07-13T06:37:46.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:46.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:46.174 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:46.220 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:46.221 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:46.505 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.71.5"},{"key":1,"val":"00000000004.68.1"},{"key":2,"val":"00000000007.65.5"},{"key":3,"val":"00000000004.93.1"},{"key":4,"val":"00000000006.50.5"},{"key":5,"val":"00000000004.90.1"},{"key":6,"val":"00000000004.60.1"},{"key":7,"val":"00000000006.73.5"},{"key":8,"val":"00000000004.48.1"},{"key":9,"val":"00000000004.93.1"},{"key":10,"val":"00000000004.70.1"}]} 2020-07-13T06:37:46.506 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-63 2020-07-13T06:37:46.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:46.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:46.507 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:46.554 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:46.555 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:46.789 INFO:teuthology.orchestra.run.smithi014.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.71.5","timestamp":"2020-07-13T06:37:13.853241Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1","timestamp":"2020-07-13T06:37:23.163134Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.65.5","timestamp":"2020-07-13T06:37:13.798400Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1","timestamp":"2020-07-13T06:37:23.094168Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.50.5","timestamp":"2020-07-13T06:37:13.748139Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.90.1","timestamp":"2020-07-13T06:37:23.163182Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1","timestamp":"2020-07-13T06:37:23.086147Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.73.5","timestamp":"2020-07-13T06:37:13.902998Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1","timestamp":"2020-07-13T06:37:23.094159Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1","timestamp":"2020-07-13T06:37:23.163157Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1","timestamp":"2020-07-13T06:37:23.094227Z"}}}] 2020-07-13T06:37:46.790 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=cngrzz-63 2020-07-13T06:37:46.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:46.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:46.793 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:46.842 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:46.842 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:47.098 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.71.5"},{"key":1,"val":"00000000004.68.1"},{"key":2,"val":"00000000007.65.5"},{"key":3,"val":"00000000004.93.1"},{"key":4,"val":"00000000006.50.5"},{"key":5,"val":"00000000004.90.1"},{"key":6,"val":"00000000004.60.1"},{"key":7,"val":"00000000006.73.5"},{"key":8,"val":"00000000004.48.1"},{"key":9,"val":"00000000004.93.1"},{"key":10,"val":"00000000004.70.1"}]} 2020-07-13T06:37:47.099 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-63 2020-07-13T06:37:47.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:47.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:47.101 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:47.190 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:47.191 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:47.472 INFO:teuthology.orchestra.run.smithi017.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.71.5","timestamp":"2020-07-13T06:37:13.853241Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1","timestamp":"2020-07-13T06:37:23.047213Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.65.5","timestamp":"2020-07-13T06:37:13.798400Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1","timestamp":"2020-07-13T06:37:23.096321Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.50.5","timestamp":"2020-07-13T06:37:13.748139Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.90.1","timestamp":"2020-07-13T06:37:23.047348Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1","timestamp":"2020-07-13T06:37:23.253202Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.73.5","timestamp":"2020-07-13T06:37:13.902998Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1","timestamp":"2020-07-13T06:37:23.141259Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1","timestamp":"2020-07-13T06:37:23.047375Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1","timestamp":"2020-07-13T06:37:23.253233Z"}}}] 2020-07-13T06:37:47.473 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=cngrzz-63 2020-07-13T06:37:47.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:47.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'cngrzz-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:47.475 INFO:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket cngrzz-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:47.524 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:47.525 INFO:teuthology.orchestra.run.smithi017.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T06:37:47.772 INFO:teuthology.orchestra.run.smithi017.stdout:{"markers":[{"key":0,"val":"00000000006.71.5"},{"key":1,"val":"00000000004.68.1"},{"key":2,"val":"00000000007.65.5"},{"key":3,"val":"00000000004.93.1"},{"key":4,"val":"00000000006.50.5"},{"key":5,"val":"00000000004.90.1"},{"key":6,"val":"00000000004.60.1"},{"key":7,"val":"00000000006.73.5"},{"key":8,"val":"00000000004.48.1"},{"key":9,"val":"00000000004.93.1"},{"key":10,"val":"00000000004.70.1"}]} 2020-07-13T06:37:47.772 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=cngrzz-63 2020-07-13T06:37:47.773 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:47.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'cngrzz-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-13T06:37:47.775 INFO:teuthology.orchestra.run.smithi014:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket cngrzz-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-13T06:37:47.820 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setuser ceph since I am not root 2020-07-13T06:37:47.821 INFO:teuthology.orchestra.run.smithi014.stderr:ignoring --setgroup ceph since I am not root 2020-07-13T17:50:43.364 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-07-13T17:50:43.408 DEBUG:teuthology.task.console_log:Killing console logger for smithi017 2020-07-13T17:50:43.409 DEBUG:teuthology.task.console_log:Killing console logger for smithi014 2020-07-13T17:50:43.410 DEBUG:teuthology.task.console_log:Killing console logger for smithi017 2020-07-13T17:50:43.410 DEBUG:teuthology.task.console_log:Killing console logger for smithi014 2020-07-13T17:50:43.412 DEBUG:teuthology.exit:Finished running handlers